Plants Seedlings Classification

Context

In recent times, the field of agriculture has been in urgent need of modernizing, since the amount of manual work people need to put in to check if plants are growing correctly is still highly extensive. Despite several advances in agricultural technology, people working in the agricultural industry still need to have the ability to sort and recognize different plants and weeds, which takes a lot of time and effort in the long term.

The potential is ripe for this trillion-dollar industry to be greatly impacted by technological innovations that cut down on the requirement for manual labor, and this is where Artificial Intelligence can benefit the workers in this field, as the time and energy required to identify plant seedlings will be greatly shortened by the use of AI and Deep Learning. The ability to do so far more efficiently and even more effectively than experienced manual labor could lead to better crop yields, the freeing up of human involvement for higher-order agricultural decision making, and in the long term will result in more sustainable environmental practices in agriculture as well.

AI Generated Image

Objective

To classify the plant seedlings into their respective categories.

Dataset

The Aarhus University Signal Processing group, in collaboration with the University of Southern Denmark, has provided the data containing images of unique plants belonging to 12 different species.

There are 12 plant species (categories) in the dataset:

• Black-grass
• Charlock
• Cleavers
• Common Chickweed
• Common Wheat
• Fat Hen
• Loose Silky-bent
• Maize
• Scentless Mayweed
• Shepherds Purse
• Small-flowered Cranesbill
• Sugar beet

Setup

Setup Google Drive

#connect to google drive
  from google.colab import drive
  drive.mount('/content/drive')
  

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
  

#parent directory where images are stored
  image_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_images.npy' #npy file containing images
  label_file = '/content/drive/MyDrive/Projects/ML/Data/plantSeedlings_Labels.csv' #csv file containing the labels for each image of the npy file
  

Import Libraries and Basics

#visualkeras is a library to visualize cnn architectures
  !pip install visualkeras
  

Requirement already satisfied: visualkeras in /usr/local/lib/python3.10/dist-packages (0.0.2)
  Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (9.4.0)
  Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.23.5)
  Requirement already satisfied: aggdraw>=1.3.11 in /usr/local/lib/python3.10/dist-packages (from visualkeras) (1.3.16)
  

#library for creating data paths
  import os

  #library for randomly selecting data points
  import random

  #library for performing numerical computations
  import numpy as np

  #library for data manipulation and analysis
  import pandas as pd

  #library for data visualization
  import seaborn as sns

  #library for creating and showing data
  import matplotlib.pyplot as plt

  #library for reading and showing images
  import matplotlib.image as mpimg

  #Python Imaging Library to handle image files
  from PIL import Image

  #OpenCv for image and video processing
  import cv2

  #sklearn
  from sklearn.model_selection import train_test_split
  from sklearn.utils import class_weight
  from sklearn.utils.class_weight import compute_sample_weight
  from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
  from sklearn.preprocessing import LabelBinarizer

  #tensorflow
  import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras.preprocessing.image import img_to_array, load_img

  #importing all the required sub-modules from Keras
  from keras import backend
  from keras.utils import to_categorical
  from keras.models import Sequential, Model
  from keras.applications.vgg16 import VGG16
  from keras.preprocessing.image import ImageDataGenerator

  from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, BatchNormalization, Dropout
  from keras import losses, optimizers
  from keras.optimizers import RMSprop,Adam,SGD #optimiers for optimizing the model
  from keras.callbacks import EarlyStopping, ModelCheckpoint  #regularization method to prevent the overfitting
  from keras.utils.vis_utils import plot_model

  #library to visualize model architecture
  import visualkeras
  

#ignore warnings that may occur due to deprecations
  import warnings
  warnings.filterwarnings("ignore")
  

Data Analysis and Data Treatment

First we will load the image_files and their corresponding labels.

#load the numpy array
  img_array = np.load(image_file)
  

#visualize contents of array, index 0
  img_array[0]
  

array([[[ 35,  52,  78],
          [ 36,  49,  76],
          [ 31,  45,  69],
          ...,
          [ 78,  95, 114],
          [ 76,  93, 110],
          [ 80,  95, 109]],

         [[ 33,  46,  68],
          [ 37,  50,  73],
          [ 48,  65,  83],
          ...,
          [ 81,  96, 113],
          [ 74,  89, 105],
          [ 83,  95, 109]],

         [[ 34,  50,  68],
          [ 35,  52,  72],
          [ 70,  85, 101],
          ...,
          [ 83,  97, 112],
          [ 79,  94, 108],
          [ 79,  94, 107]],

         ...,

         [[ 35,  50,  69],
          [ 42,  57,  73],
          [ 42,  57,  72],
          ...,
          [ 60,  76,  92],
          [ 67,  81,  97],
          [ 64,  77,  95]],

         [[ 36,  52,  67],
          [ 48,  63,  78],
          [ 41,  57,  73],
          ...,
          [ 44,  66,  83],
          [ 58,  76,  91],
          [ 57,  74,  90]],

         [[ 44,  58,  70],
          [ 43,  57,  73],
          [ 40,  55,  72],
          ...,
          [ 41,  70,  92],
          [ 55,  78,  97],
          [ 61,  79,  96]]], dtype=uint8)

Shape of Array images

#shape of the image array
  img_array.shape
  

(4750, 128, 128, 3)

The image array contains 4,750 records. Each record has a length of 128 rows and 128 columns (resolution in pixels). And each element has three values representing Red, Green, Blue (RGB).

#read the labels and store in a dataframe
  label_df = pd.read_csv(label_file)
  

#display the head of the labels dataset
  label_df.head(10)
  

	Label
0	Small-flowered Cranesbill
1	Small-flowered Cranesbill
2	Small-flowered Cranesbill
3	Small-flowered Cranesbill
4	Small-flowered Cranesbill
5	Small-flowered Cranesbill
6	Small-flowered Cranesbill
7	Small-flowered Cranesbill
8	Small-flowered Cranesbill
9	Small-flowered Cranesbill

Shape of dataframe Labels

#print shape of label dataframe
  label_df.shape
  

(4750, 1)

There are 4750 records in total.

#display information of the label dataframe and its data type
  label_df['Label'].info
  

<bound method Series.info of 0       Small-flowered Cranesbill
  1       Small-flowered Cranesbill
  2       Small-flowered Cranesbill
  3       Small-flowered Cranesbill
  4       Small-flowered Cranesbill
                    ...
  4745             Loose Silky-bent
  4746             Loose Silky-bent
  4747             Loose Silky-bent
  4748             Loose Silky-bent
  4749             Loose Silky-bent
  Name: Label, Length: 4750, dtype: object>

We should convert 'Label' to a string type because later we will convert it to one-hot vectors. To convert to one-hot vectors, we will use 'to_categorical' which expects an array of integers or numeric values. Otherwise type may throw an error.

#convert to string type
  label_df['Label'] = label_df['Label'].astype(str)
  

Number of Unique Classes

#Number of unique classes (labels)
  num_classes = label_df.nunique()[0]
  print("The number of seed classes is ",num_classes)
  

The number of seed classes is  12
  

Name of Classes

#create an array containing the labels of classes
  name_classes = label_df['Label'].unique()
  name_classes
  

array(['Small-flowered Cranesbill', 'Fat Hen', 'Shepherds Purse',
         'Common wheat', 'Common Chickweed', 'Charlock', 'Cleavers',
         'Scentless Mayweed', 'Sugar beet', 'Maize', 'Black-grass',
         'Loose Silky-bent'], dtype=object)

Distribution of Classes

#visualize the seedlings label
  figsize=(14, 8)
  count_plot = sns.countplot(data=label_df, x="Label")

  #rotate x-axis labels
  count_plot.set_xticklabels(count_plot.get_xticklabels(), rotation=90)
  count_plot.set(xlabel='seed')

  plt.show()
  

#Number of items per categories
  label_df.value_counts()
  

Label
  Loose Silky-bent             654
  Common Chickweed             611
  Scentless Mayweed            516
  Small-flowered Cranesbill    496
  Fat Hen                      475
  Charlock                     390
  Sugar beet                   385
  Cleavers                     287
  Black-grass                  263
  Shepherds Purse              231
  Common wheat                 221
  Maize                        221
  dtype: int64

The data is imbalanced. Class Maize and Common Wheat are roughly 1/3 the size of Loose Silky-bent.

Plotting images

#convert dataframe to array
  label_array = label_df.to_numpy().squeeze()
  

Randomly from the entire set

#Displaying images randomly

  #Generate random indices to select rows x columns images
  rows = 4
  cols = 5

  ix_img = np.random.choice(len(label_array), size=rows*cols, replace=False)

  # Select the images and labels
  images = img_array[ix_img]
  labels = label_array[ix_img].tolist()

  # Define the size of the figure
  fig = plt.figure(figsize=(12, 12))

  # Create a rows x cols grid of subplots
  for i in range(rows*cols):
      ax = fig.add_subplot(rows, cols, i+1)
      ax.imshow(images[i])
      ax.set_xlabel(str(labels[i]), fontsize=10)
      ax.set_xticks([])
      ax.set_yticks([])

  # Show the figure
  plt.tight_layout()
  plt.subplots_adjust(wspace=0.2)
  plt.show()
  

Randomly for each class

#Displaying images (randomly) from each category

  num_images = 5

  # Create a dictionary to store the indices of each category
  category_indices = {}
  for category in range(num_classes):
      category_indices[category] = label_df[label_df['Label'] == category].index.tolist()

  # Loop through each unique label
  for label in name_classes:

      # Get the indices of all images with the current label
      indices = np.where(label_df['Label'] == label)[0]

      # Choose num_images random indices
      random_indices = np.random.choice(indices, size=num_images, replace=False)

      # Get the corresponding images from "img_array"
      images = img_array[random_indices]

      # Create a plot of the images
      fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
      for i, image in enumerate(images):
          ax[i].imshow(image)
          ax[i].axis('off')

      # Set the suptitle and adjust the spacing
      fig.suptitle(label, fontsize=16)
      fig.subplots_adjust(top=1.1, bottom=0.1)
      plt.show()
  

Output hidden; open in https://colab.research.google.com to view.

Observations from EDA

1. There are 12 unique classes.
2. There are 4,750 records.
3. The classes are imbalanced as seen in the distribution plot.
4. The class "Loose Silky-bent" is rougly three times larger than the class "Maize" or "Common wheat"
5. The images size is 128 x 128 pixes, with 3 channels (RGB).
6. There are images that are out of focus.
7. Some images are dark.
8. Some images contain bar codes, stones, etc.
9. Some images also have random variations that may be due to camera sensor noise, compression artifacts, image processing, user error, etc.
10. Most seedlings are green against a blueshish rock-soil which do not contrast well with the dark green color of most seedlings.

From the dataset image analysis we foresee the need to remove noise from the images. To do this we can use Gaussian Blurring.

Remove Noise using Gaussian Blurring

We will perform Gaussian Blurring to smooth the images by averaging the intensity of each pixel with the intensity of neighboring pixels.

# Choose a random index for the image
  index = np.random.randint(0, img_array.shape[0])

  # Extract the image and its label
  image = img_array[index, :, :, :]
  label = label_df.iloc[index, 0]

  fig = plt.figure(figsize=(18, 18))

  # Plot the original image
  plt.subplot(1, 2, 1)
  plt.imshow(image.squeeze(), cmap='gray')
  plt.title('Original Image (Label: {})'.format(label))
  plt.axis('off')
  

(-0.5, 127.5, 127.5, -0.5)

# Define the kernel size for the Gaussian blur
  kernel_size = (3, 3)

  # Apply Gaussian blur to each image in the img_array
  for i in range(img_array.shape[0]):
      img = img_array[i,:,:,:]
      img = cv2.GaussianBlur(img, kernel_size, 0)
      img_array[i,:,:,:] = img
  

Apply Normalization

After Gaussian Blurring we will scale the pixel values to a fixed range (i.e. 0 to 1).

# Convert the data type of the img_array to float32
  img_array = img_array.astype('float32')

  # Normalize the pixel values to the range of 0 to 1
  img_array = img_array / 255.0
  

# Plot the preprocessed image

  fig = plt.figure(figsize=(18, 18))
  plt.subplot(1, 2, 2)
  plt.imshow(image.squeeze(), cmap='gray')
  plt.title('Preprocessed Image (Label: {})'.format(label))
  plt.axis('off')

  # Show the plots
  plt.show()
  

Images after Gaussian Blurring and Normalization

#Displaying images (randomly) from each category

  num_images = 5

  # Create a dictionary to store the indices of each category
  category_indices = {}
  for category in range(num_classes):
      category_indices[category] = label_df[label_df['Label'] == category].index.tolist()

  # Loop through each unique label
  for label in name_classes:

      # Get the indices of all images with the current label
      indices = np.where(label_df['Label'] == label)[0]

      # Choose num_images random indices
      random_indices = np.random.choice(indices, size=num_images, replace=False)

      # Get the corresponding images from "img_array"
      images = img_array[random_indices]

      # Create a plot of the images
      fig, ax = plt.subplots(1, num_images, figsize=(20, 4))
      for i, image in enumerate(images):
          ax[i].imshow(image)
          ax[i].axis('off')

      # Set the suptitle and adjust the spacing
      fig.suptitle(label, fontsize=16)
      fig.subplots_adjust(top=1.1, bottom=0.1)
      plt.show()
  

Output hidden; open in https://colab.research.google.com to view.

Beware that the differences in the appearance of the original image and the Gaussian-Normalized image are hard to notice by a human eye.

Define a Dictionary to map labels to integer values

# Define a dictionary to map each label to a unique integer value

  label_map = {'Small-flowered Cranesbill':0, 'Fat Hen':1, 'Shepherds Purse':2,
         'Common wheat':3, 'Common Chickweed':4, 'Charlock':5, 'Cleavers':6,
         'Scentless Mayweed':7, 'Sugar beet':8, 'Maize':9, 'Black-grass':10,
         'Loose Silky-bent':11}

  # Convert the label strings to integer values using the label map
  label_df['Label'] = label_df['Label'].map(label_map)
  

Splitting Data

#splitting data into training and testing set
  X_train, X_test, y_train, y_test = train_test_split(img_array,label_df['Label'], test_size=0.1, random_state=42, stratify=label_df['Label'])
  

Treating Class Imbalance

We can oversample the minority class by adding more samples to that minority classes doing augmentation, generating synthetic data, etc. We can undersample the majority class by removing samples from the majority class to balance the distribution. However we will lose information and although the distribution of the majority class to minitory class is 3 to 1, this may not be a good approach because our dataset is not very large. We can also use data augmentation. And we can also increase the weights of the minority classes.

The data is imbalanced as we have seen earlier. To avoid bias toward the majority class, or underfitting the minority class, we should balance the data.

Balancing the data on the Training set by adjusting weights.

The main reason to use this technique is because reprocessing the images have a very high computational cost.

# Calculate class weights to balance the data
  class_weights = {}
  for i in range(12):
      class_weights[i] = np.sum(y_train == i) / len(y_train)

  print(class_weights)
  

{0: 0.10432748538011696, 1: 0.09988304093567252, 2: 0.048654970760233916, 3: 0.04654970760233918, 4: 0.1286549707602339, 5: 0.08210526315789474, 6: 0.060350877192982454, 7: 0.10853801169590643, 8: 0.08116959064327485, 9: 0.04654970760233918, 10: 0.05543859649122807, 11: 0.13777777777777778}
  

# Use the class weights to balance the data
  sample_weights = compute_sample_weight(class_weight=class_weights, y=y_train)
  

Make Data Compatible - One hot Encoding

y_train.info
  

<bound method Series.info of 437      0
  1571     4
  838      1
  4040    10
  4707    11
          ..
  746      1
  1217     3
  3058     7
  4682    11
  787      1
  Name: Label, Length: 4275, dtype: int64>

# Convert the labels to one-hot vector (we only encode the y_train and y_test because these are categorical)
  y_train_onehot = to_categorical(y_train, num_classes=12)
  y_test_onehot = to_categorical(y_test, num_classes=12)
  

print("X_train shape:",X_train.shape)
  print("X_test shape:",X_test.shape)

  print("y_train shape:",y_train.shape)
  print("y_test shape:",y_test.shape)

  print("y_train one-hot encoded shape:",y_train_onehot.shape)
  print("y_test one-hot encoded shape:",y_test_onehot.shape)
  

X_train shape: (4275, 128, 128, 3)
  X_test shape: (475, 128, 128, 3)
  y_train shape: (4275,)
  y_test shape: (475,)
  y_train one-hot encoded shape: (4275, 12)
  y_test one-hot encoded shape: (475, 12)
  

The data is compatible with Keras. img_array is a 4D array with shape (4750, 128, 128, 3), with 4750 images, sized 128x128px and the 3 RGB channels.

We already one hot encoded y_train and y_test, so these are also compatible with Keras.

Convolutional Neural Network Models

input_shape = (128,128,3)
  

#create dataframe to store scores
  Scoresdf = pd.DataFrame(columns =['model', 'accuracy', 'layers', 'total parameters', 'neurons', 'optimizer'])
  

We will first start with a simple model of only 1 Convolution layer.

Model 0

Model Definition

Layers : 4

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: MaxPooling2D layer

Layer 3: Flatten layer

Layer 4: Dense layer : 12 neurons, Relu

Optimizer: Adam

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

# Define the model architecture
  model0 = Sequential()

  #Convolution with 32 filters size 3,3 and relu
  model0.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))

  #add max pooling to reduce the size of output of first convolution layer
  model0.add(MaxPooling2D(pool_size=(2, 2)))

  #reshape the output to a flat vector
  model0.add(Flatten())

  #add output layer
  model0.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model0.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model0.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   flatten (Flatten)           (None, 127008)            0

   dense (Dense)               (None, 12)                1524108

  =================================================================
  Total params: 1,525,004
  Trainable params: 1,525,004
  Non-trainable params: 0
  _________________________________________________________________
  

Model Visualization

#visualize the CNN architecture
  plot_model(model0, to_file='model0_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize the CNN architecture using visualkeras
  visualkeras.layered_view(model0, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 30 epochs and validation_split as 10%
  history0 = model0.fit(X_train, y_train_onehot, epochs=30, batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/30
  120/121 [============================>.] - ETA: 0s - loss: 1.7592 - accuracy: 0.4187
  Epoch 1: val_accuracy improved from -inf to 0.49533, saving model to best_model.h5
  121/121 [==============================] - 5s 22ms/step - loss: 1.7579 - accuracy: 0.4195 - val_loss: 1.4393 - val_accuracy: 0.4953
  Epoch 2/30
  117/121 [============================>.] - ETA: 0s - loss: 1.1295 - accuracy: 0.6242
  Epoch 2: val_accuracy improved from 0.49533 to 0.52804, saving model to best_model.h5
  121/121 [==============================] - 2s 18ms/step - loss: 1.1295 - accuracy: 0.6249 - val_loss: 1.3444 - val_accuracy: 0.5280
  Epoch 3/30
  118/121 [============================>.] - ETA: 0s - loss: 0.8919 - accuracy: 0.7076
  Epoch 3: val_accuracy improved from 0.52804 to 0.56075, saving model to best_model.h5
  121/121 [==============================] - 2s 13ms/step - loss: 0.8918 - accuracy: 0.7065 - val_loss: 1.3244 - val_accuracy: 0.5607
  Epoch 4/30
  117/121 [============================>.] - ETA: 0s - loss: 0.7481 - accuracy: 0.7591
  Epoch 4: val_accuracy improved from 0.56075 to 0.61449, saving model to best_model.h5
  121/121 [==============================] - 2s 13ms/step - loss: 0.7480 - accuracy: 0.7593 - val_loss: 1.1457 - val_accuracy: 0.6145
  Epoch 5/30
  119/121 [============================>.] - ETA: 0s - loss: 0.6163 - accuracy: 0.8120
  Epoch 5: val_accuracy improved from 0.61449 to 0.65421, saving model to best_model.h5
  121/121 [==============================] - 2s 14ms/step - loss: 0.6152 - accuracy: 0.8123 - val_loss: 1.1489 - val_accuracy: 0.6542
  Epoch 6/30
  118/121 [============================>.] - ETA: 0s - loss: 0.4893 - accuracy: 0.8620
  Epoch 6: val_accuracy improved from 0.65421 to 0.65654, saving model to best_model.h5
  121/121 [==============================] - 1s 12ms/step - loss: 0.4904 - accuracy: 0.8617 - val_loss: 1.1100 - val_accuracy: 0.6565
  Epoch 7/30
  117/121 [============================>.] - ETA: 0s - loss: 0.3744 - accuracy: 0.8993
  Epoch 7: val_accuracy did not improve from 0.65654
  121/121 [==============================] - 1s 11ms/step - loss: 0.3717 - accuracy: 0.8999 - val_loss: 1.2097 - val_accuracy: 0.6262
  Epoch 8/30
  117/121 [============================>.] - ETA: 0s - loss: 0.3120 - accuracy: 0.9241
  Epoch 8: val_accuracy improved from 0.65654 to 0.66121, saving model to best_model.h5
  121/121 [==============================] - 1s 12ms/step - loss: 0.3107 - accuracy: 0.9244 - val_loss: 1.1094 - val_accuracy: 0.6612
  Epoch 9/30
  119/121 [============================>.] - ETA: 0s - loss: 0.2301 - accuracy: 0.9485
  Epoch 9: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 11ms/step - loss: 0.2295 - accuracy: 0.9488 - val_loss: 1.2245 - val_accuracy: 0.6589
  Epoch 10/30
  117/121 [============================>.] - ETA: 0s - loss: 0.1703 - accuracy: 0.9682
  Epoch 10: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.1722 - accuracy: 0.9670 - val_loss: 1.2595 - val_accuracy: 0.6075
  Epoch 11/30
  118/121 [============================>.] - ETA: 0s - loss: 0.1336 - accuracy: 0.9783
  Epoch 11: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 11ms/step - loss: 0.1342 - accuracy: 0.9782 - val_loss: 1.2517 - val_accuracy: 0.6425
  Epoch 12/30
  117/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9904
  Epoch 12: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.0926 - accuracy: 0.9906 - val_loss: 1.3501 - val_accuracy: 0.6145
  Epoch 13/30
  119/121 [============================>.] - ETA: 0s - loss: 0.0705 - accuracy: 0.9937
  Epoch 13: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.0708 - accuracy: 0.9938 - val_loss: 1.2872 - val_accuracy: 0.6495
  Epoch 14/30
  116/121 [===========================>..] - ETA: 0s - loss: 0.0489 - accuracy: 0.9976
  Epoch 14: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 2s 13ms/step - loss: 0.0483 - accuracy: 0.9977 - val_loss: 1.3812 - val_accuracy: 0.6379
  Epoch 15/30
  116/121 [===========================>..] - ETA: 0s - loss: 0.0337 - accuracy: 0.9992
  Epoch 15: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 2s 13ms/step - loss: 0.0335 - accuracy: 0.9992 - val_loss: 1.3267 - val_accuracy: 0.6449
  Epoch 16/30
  118/121 [============================>.] - ETA: 0s - loss: 0.0285 - accuracy: 0.9981
  Epoch 16: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.0285 - accuracy: 0.9982 - val_loss: 1.3954 - val_accuracy: 0.6519
  Epoch 17/30
  120/121 [============================>.] - ETA: 0s - loss: 0.0229 - accuracy: 1.0000
  Epoch 17: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.0229 - accuracy: 1.0000 - val_loss: 1.4152 - val_accuracy: 0.6449
  Epoch 18/30
  121/121 [==============================] - ETA: 0s - loss: 0.0156 - accuracy: 1.0000
  Epoch 18: val_accuracy did not improve from 0.66121
  121/121 [==============================] - 1s 12ms/step - loss: 0.0156 - accuracy: 1.0000 - val_loss: 1.4935 - val_accuracy: 0.6379
  Epoch 18: early stopping
  

Notice that at Epoch 16 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.65654 and validation loss 1.2791.

Model0 - Accuracy vs Epoch curve

plt.plot(history0.history['accuracy'])
  plt.plot(history0.history['val_accuracy'])
  plt.title('model0 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

We can see that this model performs poorly. The model performs very well on training but very poorly in the validation. It is clearly overfitting.

Evaluation of model0 model.evaluate using the Test data

# Evaluate the model on the test data
  score0 = model0.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score0[0])
  print('Test accuracy:', score0[1])
  

Test loss: 1.6907126903533936
  Test accuracy: 0.648421049118042
  

# Test Prediction
  y_pred = model0.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 6ms/step
  

# Test Accuracy
  accuracyScore0 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore0)
  

0.6484210526315789
  

Classification Report

# Compute the classification report
  cr0 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr0)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.76      0.84      0.80        50
                    Fat Hen       0.61      0.62      0.62        48
            Shepherds Purse       0.42      0.22      0.29        23
               Common wheat       0.44      0.32      0.37        22
           Common Chickweed       0.93      0.67      0.78        61
                   Charlock       0.83      0.97      0.89        39
                   Cleavers       0.68      0.52      0.59        29
          Scentless Mayweed       0.52      0.81      0.63        52
                 Sugar beet       0.64      0.47      0.55        38
                      Maize       0.59      0.59      0.59        22
                Black-grass       0.36      0.38      0.37        26
           Loose Silky-bent       0.65      0.72      0.69        65

                   accuracy                           0.65       475
                  macro avg       0.62      0.60      0.60       475
               weighted avg       0.66      0.65      0.64       475

  

Charlock has a higher precision, recall, and F1 score (0.88, 90, 0.89) than all other classes. Black-grass has the opposite (all low: 0.36, 0.46, 0.41).

Model0 has an overall accuracy of 0.65, it correctly predicted the seedling's class for 65% of the images in the test set.

The precision, recall, and F1-score for each class vary, with some classes having good performance (e.g., Charlock) and others having poor performance (e.g., Shepherds Purse).

The macro average F1 score is 0.65, which means that Model0 is not so good overall. The weighted average F1 score is also .65 and this means that the model is not biased towards any particular class.

Scores_Models_df = Scoresdf.append({'model':'Model0','accuracy':accuracyScore0,'layers':'4','total parameters':'1,525,004', 'neurons':'12', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Set the confusion matrix
  cm0 = confusion_matrix(normal_y_test, y_test_pred_classes)

  plt.figure(figsize=(8,6))

  sns.heatmap(cm0, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model0 is able to predict some classes with good accuracy but others are not predicted well. We can build better models to improve the performance on the other classes.

This model trained 1,525,004 parameters.

We called our first model, Model0 because this was a simple model with only one Convolution layer, so we didn't expect much from it.

Model 1

Model Definition

This model builds on Model0. Model1 adds a second Convolutional layer and the fully connected layer has 128 neurons to improve the previous Model (Model0).

• Layers : 8
• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu
• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Flatten layer
• Layer 6: Dense layer : 128 neurons, Relu

• Layer 7: Droput layer: 0.5

• Optimizer: Adam

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# Define the model architecture
  model1 = Sequential()

  #Convolution with 32 filters size 3,3 and relu
  model1.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))

  #add max pooling to reduce the size of output of first convolution layer
  model1.add(MaxPooling2D(pool_size=(2, 2)))

  #Convolution with 64 filters size 3,3 and relu
  model1.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))

  #add max pooling to reduce the size of output of second convolution layer
  model1.add(MaxPooling2D(pool_size=(2, 2)))

  #reshape the output to a flat vector
  model1.add(Flatten())

  #fully connected layer with 128 neurons
  model1.add(Dense(128, activation='relu'))

  #add dropout to prevent overfitting
  model1.add(Dropout(0.5))

  #add output layer
  model1.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model1.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0
   2D)

   flatten (Flatten)           (None, 57600)             0

   dense (Dense)               (None, 128)               7372928

   dropout (Dropout)           (None, 128)               0

   dense_1 (Dense)             (None, 12)                1548

  =================================================================
  Total params: 7,393,868
  Trainable params: 7,393,868
  Non-trainable params: 0
  _________________________________________________________________
  

Model Visualization

#visualize the CNN architecture
  plot_model(model1, to_file='model1_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize the CNN architecture using visualkeras
  visualkeras.layered_view(model1, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 60 epochs and validation_split as 10%
  history1 = model1.fit(X_train,
            y_train_onehot,
            epochs=60,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/60
  119/121 [============================>.] - ETA: 0s - loss: 1.9557 - accuracy: 0.3456
  Epoch 1: val_accuracy improved from -inf to 0.39953, saving model to best_model.h5
  121/121 [==============================] - 4s 24ms/step - loss: 1.9516 - accuracy: 0.3473 - val_loss: 1.6871 - val_accuracy: 0.3995
  Epoch 2/60
  118/121 [============================>.] - ETA: 0s - loss: 1.4500 - accuracy: 0.5085
  Epoch 2: val_accuracy improved from 0.39953 to 0.57009, saving model to best_model.h5
  121/121 [==============================] - 3s 24ms/step - loss: 1.4500 - accuracy: 0.5095 - val_loss: 1.2356 - val_accuracy: 0.5701
  Epoch 3/60
  118/121 [============================>.] - ETA: 0s - loss: 1.2128 - accuracy: 0.5879
  Epoch 3: val_accuracy improved from 0.57009 to 0.59813, saving model to best_model.h5
  121/121 [==============================] - 3s 24ms/step - loss: 1.2120 - accuracy: 0.5885 - val_loss: 1.2166 - val_accuracy: 0.5981
  Epoch 4/60
  118/121 [============================>.] - ETA: 0s - loss: 1.0462 - accuracy: 0.6496
  Epoch 4: val_accuracy improved from 0.59813 to 0.63785, saving model to best_model.h5
  121/121 [==============================] - 3s 23ms/step - loss: 1.0478 - accuracy: 0.6483 - val_loss: 1.0640 - val_accuracy: 0.6379
  Epoch 5/60
  118/121 [============================>.] - ETA: 0s - loss: 0.9670 - accuracy: 0.6700
  Epoch 5: val_accuracy improved from 0.63785 to 0.66121, saving model to best_model.h5
  121/121 [==============================] - 2s 21ms/step - loss: 0.9692 - accuracy: 0.6691 - val_loss: 1.0265 - val_accuracy: 0.6612
  Epoch 6/60
  121/121 [==============================] - ETA: 0s - loss: 0.8473 - accuracy: 0.7024
  Epoch 6: val_accuracy improved from 0.66121 to 0.67523, saving model to best_model.h5
  121/121 [==============================] - 5s 39ms/step - loss: 0.8473 - accuracy: 0.7024 - val_loss: 0.9550 - val_accuracy: 0.6752
  Epoch 7/60
  120/121 [============================>.] - ETA: 0s - loss: 0.7730 - accuracy: 0.7372
  Epoch 7: val_accuracy did not improve from 0.67523
  121/121 [==============================] - 2s 20ms/step - loss: 0.7724 - accuracy: 0.7375 - val_loss: 0.9919 - val_accuracy: 0.6589
  Epoch 8/60
  120/121 [============================>.] - ETA: 0s - loss: 0.6773 - accuracy: 0.7633
  Epoch 8: val_accuracy improved from 0.67523 to 0.71729, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.6779 - accuracy: 0.7627 - val_loss: 0.9073 - val_accuracy: 0.7173
  Epoch 9/60
  120/121 [============================>.] - ETA: 0s - loss: 0.5863 - accuracy: 0.7974
  Epoch 9: val_accuracy did not improve from 0.71729
  121/121 [==============================] - 2s 18ms/step - loss: 0.5861 - accuracy: 0.7975 - val_loss: 0.9788 - val_accuracy: 0.7033
  Epoch 10/60
  119/121 [============================>.] - ETA: 0s - loss: 0.5086 - accuracy: 0.8262
  Epoch 10: val_accuracy improved from 0.71729 to 0.72196, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 0.5104 - accuracy: 0.8251 - val_loss: 0.9273 - val_accuracy: 0.7220
  Epoch 11/60
  119/121 [============================>.] - ETA: 0s - loss: 0.4790 - accuracy: 0.8290
  Epoch 11: val_accuracy improved from 0.72196 to 0.72664, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 0.4803 - accuracy: 0.8290 - val_loss: 0.9444 - val_accuracy: 0.7266
  Epoch 12/60
  121/121 [==============================] - ETA: 0s - loss: 0.4102 - accuracy: 0.8568
  Epoch 12: val_accuracy improved from 0.72664 to 0.73131, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.4102 - accuracy: 0.8568 - val_loss: 0.9420 - val_accuracy: 0.7313
  Epoch 13/60
  118/121 [============================>.] - ETA: 0s - loss: 0.3798 - accuracy: 0.8673
  Epoch 13: val_accuracy improved from 0.73131 to 0.74766, saving model to best_model.h5
  121/121 [==============================] - 3s 23ms/step - loss: 0.3841 - accuracy: 0.8661 - val_loss: 0.9019 - val_accuracy: 0.7477
  Epoch 14/60
  121/121 [==============================] - ETA: 0s - loss: 0.3665 - accuracy: 0.8742
  Epoch 14: val_accuracy improved from 0.74766 to 0.75234, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.3665 - accuracy: 0.8742 - val_loss: 0.9895 - val_accuracy: 0.7523
  Epoch 15/60
  118/121 [============================>.] - ETA: 0s - loss: 0.3009 - accuracy: 0.8938
  Epoch 15: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 18ms/step - loss: 0.2996 - accuracy: 0.8945 - val_loss: 0.9900 - val_accuracy: 0.7407
  Epoch 16/60
  121/121 [==============================] - ETA: 0s - loss: 0.2900 - accuracy: 0.8997
  Epoch 16: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 3s 21ms/step - loss: 0.2900 - accuracy: 0.8997 - val_loss: 0.9611 - val_accuracy: 0.7407
  Epoch 17/60
  120/121 [============================>.] - ETA: 0s - loss: 0.2824 - accuracy: 0.8979
  Epoch 17: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 18ms/step - loss: 0.2824 - accuracy: 0.8978 - val_loss: 1.0341 - val_accuracy: 0.7407
  Epoch 18/60
  120/121 [============================>.] - ETA: 0s - loss: 0.2479 - accuracy: 0.9146
  Epoch 18: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 19ms/step - loss: 0.2483 - accuracy: 0.9145 - val_loss: 1.1104 - val_accuracy: 0.7453
  Epoch 19/60
  120/121 [============================>.] - ETA: 0s - loss: 0.2387 - accuracy: 0.9128
  Epoch 19: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 20ms/step - loss: 0.2383 - accuracy: 0.9129 - val_loss: 1.0547 - val_accuracy: 0.7477
  Epoch 20/60
  118/121 [============================>.] - ETA: 0s - loss: 0.2062 - accuracy: 0.9266
  Epoch 20: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 20ms/step - loss: 0.2065 - accuracy: 0.9264 - val_loss: 1.0478 - val_accuracy: 0.7523
  Epoch 21/60
  119/121 [============================>.] - ETA: 0s - loss: 0.2306 - accuracy: 0.9160
  Epoch 21: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 18ms/step - loss: 0.2302 - accuracy: 0.9163 - val_loss: 1.2161 - val_accuracy: 0.7407
  Epoch 22/60
  119/121 [============================>.] - ETA: 0s - loss: 0.2000 - accuracy: 0.9280
  Epoch 22: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 18ms/step - loss: 0.2001 - accuracy: 0.9280 - val_loss: 1.1027 - val_accuracy: 0.7430
  Epoch 23/60
  118/121 [============================>.] - ETA: 0s - loss: 0.1923 - accuracy: 0.9301
  Epoch 23: val_accuracy did not improve from 0.75234
  121/121 [==============================] - 2s 18ms/step - loss: 0.1908 - accuracy: 0.9309 - val_loss: 1.1310 - val_accuracy: 0.7383
  Epoch 23: early stopping
  

Notice that at Epoch 23 the training stopped since the validation loss had continued to increase during 10 epochs. The highest validation accuracy was 0.75 and validation loss of 1.1866.

Model1 - Accuracy vs Epoch curve

plt.plot(history1.history['accuracy'])
  plt.plot(history1.history['val_accuracy'])
  plt.title('model1 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

Notice that the model shows overfitting. Past epoch 12 the validation loss doesn't reduce or begins to increase.

The model performs poorly as we see that the training accuracy is high but the validation accuracy is poor, indicating that the model is overfitting.

Evaluation of model1 model.evaluate using the Test data

# Evaluate the model on the test data
  score1 = model1.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score1[0])
  print('Test accuracy:', score1[1])
  

Test loss: 1.2217235565185547
  Test accuracy: 0.730526328086853
  

# Test Prediction
  y_pred = model1.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 7ms/step
  

# Test Accuracy
  accuracyScore1 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore1)
  

0.7305263157894737
  

Classification Report

# Compute the classification report
  cr1 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr1)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.90      0.86      0.88        50
                    Fat Hen       0.75      0.75      0.75        48
            Shepherds Purse       0.43      0.39      0.41        23
               Common wheat       0.65      0.68      0.67        22
           Common Chickweed       0.88      0.84      0.86        61
                   Charlock       0.77      0.92      0.84        39
                   Cleavers       0.83      0.69      0.75        29
          Scentless Mayweed       0.64      0.75      0.69        52
                 Sugar beet       0.80      0.63      0.71        38
                      Maize       0.57      0.73      0.64        22
                Black-grass       0.42      0.31      0.36        26
           Loose Silky-bent       0.74      0.77      0.75        65

                   accuracy                           0.73       475
                  macro avg       0.70      0.69      0.69       475
               weighted avg       0.73      0.73      0.73       475

  

The model seems to perform well for classes such as Small-flowered Cranesbill, Common Chickweed, and Charlock with high precision, recall, and F1-score.

The model performs poorly with classes like Black-grass, Shepherds Purse, and Common wheat that have low precision, recall, and F1-score.

The macro-average and weighted-average F1-score are 0.69 and 0.73, which indicates that the model has an overall decent performance.

However, this model can be improved, especially for the clasess that perform poorly.

Scoresdf = Scoresdf.append({'model':'Model1','accuracy':accuracyScore1,'layers':'8','total parameters':'7,393,868', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Set the confusion matrix
  cm1 = confusion_matrix(normal_y_test, y_test_pred_classes)

  plt.figure(figsize=(8,6))

  sns.heatmap(cm1, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model1 one performs better than Model0 but the model isn't very good either. It trains 7,393,868 parameters which is about 4 times more parameters than Model0 and although this difference is massive, Model1 doesn't have a great performance either.

Model 2

Model2 keeps growing, to try to improve on the previous model (Model1) we added more layers.

• Layers : 10

• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 6: MaxPooling2D layer
• Layer 7: Flatten layer
• Layer 8: Dense layer : 128 neurons, Relu

• Layer 9: Droput layer: 0.5

• Optimizer: Adam

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# Define the model architecture
  model2 = Sequential()

  #Convolution with 32 filters size 3,3 and relu
  model2.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))

  #add max pooling to reduce the size of output of first convolution layer
  model2.add(MaxPooling2D(pool_size=(2,2)))

  #Convolution with 64 filters size 3,3 and relu
  model2.add(Conv2D(64, kernel_size=(3,3), activation='relu'))

  #add max pooling to reduce the size of output of first convolution layer
  model2.add(MaxPooling2D(pool_size=(2,2)))

  #Convolution with 128 filters size 3,3 and relu
  model2.add(Conv2D(128, kernel_size=(3,3), activation='relu'))

  #add max pooling to reduce the size of output of first convolution layer
  model2.add(MaxPooling2D(pool_size=(2,2)))

  #reshape the output to a flat vector
  model2.add(Flatten())

  #fully connected layer with 128 neurons
  model2.add(Dense(128, activation='relu'))

  #add dropout to prevent overfitting
  model2.add(Dropout(0.5))

  #add output layer
  model2.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model2.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0
   2D)

   flatten (Flatten)           (None, 25088)             0

   dense (Dense)               (None, 128)               3211392

   dropout (Dropout)           (None, 128)               0

   dense_1 (Dense)             (None, 12)                1548

  =================================================================
  Total params: 3,306,188
  Trainable params: 3,306,188
  Non-trainable params: 0
  _________________________________________________________________
  

Model Visualization

#visualize the cnn model architecture
  plot_model(model2, to_file='model2_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize the cnn model architecture using visualkeras
  visualkeras.layered_view(model2, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 40 epochs and validation_split as 10%
  history2 = model2.fit(X_train,
            y_train_onehot,
            epochs=40,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/40
  119/121 [============================>.] - ETA: 0s - loss: 1.9732 - accuracy: 0.3369
  Epoch 1: val_accuracy improved from -inf to 0.44626, saving model to best_model.h5
  121/121 [==============================] - 6s 25ms/step - loss: 1.9721 - accuracy: 0.3379 - val_loss: 1.5570 - val_accuracy: 0.4463
  Epoch 2/40
  118/121 [============================>.] - ETA: 0s - loss: 1.4752 - accuracy: 0.4923
  Epoch 2: val_accuracy improved from 0.44626 to 0.52570, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 1.4720 - accuracy: 0.4939 - val_loss: 1.2610 - val_accuracy: 0.5257
  Epoch 3/40
  118/121 [============================>.] - ETA: 0s - loss: 1.2448 - accuracy: 0.5728
  Epoch 3: val_accuracy improved from 0.52570 to 0.61916, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 1.2429 - accuracy: 0.5734 - val_loss: 1.0807 - val_accuracy: 0.6192
  Epoch 4/40
  118/121 [============================>.] - ETA: 0s - loss: 1.0740 - accuracy: 0.6382
  Epoch 4: val_accuracy did not improve from 0.61916
  121/121 [==============================] - 2s 19ms/step - loss: 1.0733 - accuracy: 0.6382 - val_loss: 1.0088 - val_accuracy: 0.6145
  Epoch 5/40
  119/121 [============================>.] - ETA: 0s - loss: 0.9800 - accuracy: 0.6594
  Epoch 5: val_accuracy improved from 0.61916 to 0.69159, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.9808 - accuracy: 0.6584 - val_loss: 0.8978 - val_accuracy: 0.6916
  Epoch 6/40
  120/121 [============================>.] - ETA: 0s - loss: 0.8985 - accuracy: 0.6867
  Epoch 6: val_accuracy improved from 0.69159 to 0.74065, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.8982 - accuracy: 0.6868 - val_loss: 0.8329 - val_accuracy: 0.7407
  Epoch 7/40
  118/121 [============================>.] - ETA: 0s - loss: 0.8048 - accuracy: 0.7259
  Epoch 7: val_accuracy did not improve from 0.74065
  121/121 [==============================] - 2s 20ms/step - loss: 0.8049 - accuracy: 0.7255 - val_loss: 0.8244 - val_accuracy: 0.7407
  Epoch 8/40
  118/121 [============================>.] - ETA: 0s - loss: 0.7244 - accuracy: 0.7407
  Epoch 8: val_accuracy improved from 0.74065 to 0.76636, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 0.7228 - accuracy: 0.7406 - val_loss: 0.7660 - val_accuracy: 0.7664
  Epoch 9/40
  118/121 [============================>.] - ETA: 0s - loss: 0.6450 - accuracy: 0.7688
  Epoch 9: val_accuracy improved from 0.76636 to 0.77103, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 0.6472 - accuracy: 0.7687 - val_loss: 0.7315 - val_accuracy: 0.7710
  Epoch 10/40
  118/121 [============================>.] - ETA: 0s - loss: 0.6278 - accuracy: 0.7773
  Epoch 10: val_accuracy did not improve from 0.77103
  121/121 [==============================] - 2s 20ms/step - loss: 0.6289 - accuracy: 0.7757 - val_loss: 0.7808 - val_accuracy: 0.7407
  Epoch 11/40
  119/121 [============================>.] - ETA: 0s - loss: 0.5677 - accuracy: 0.7946
  Epoch 11: val_accuracy did not improve from 0.77103
  121/121 [==============================] - 3s 22ms/step - loss: 0.5674 - accuracy: 0.7952 - val_loss: 0.7369 - val_accuracy: 0.7593
  Epoch 12/40
  120/121 [============================>.] - ETA: 0s - loss: 0.5108 - accuracy: 0.8104
  Epoch 12: val_accuracy improved from 0.77103 to 0.78037, saving model to best_model.h5
  121/121 [==============================] - 3s 23ms/step - loss: 0.5105 - accuracy: 0.8105 - val_loss: 0.6926 - val_accuracy: 0.7804
  Epoch 13/40
  121/121 [==============================] - ETA: 0s - loss: 0.4677 - accuracy: 0.8227
  Epoch 13: val_accuracy improved from 0.78037 to 0.80140, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.4677 - accuracy: 0.8227 - val_loss: 0.6959 - val_accuracy: 0.8014
  Epoch 14/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4548 - accuracy: 0.8276
  Epoch 14: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 2s 20ms/step - loss: 0.4534 - accuracy: 0.8290 - val_loss: 0.7239 - val_accuracy: 0.7897
  Epoch 15/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4509 - accuracy: 0.8337
  Epoch 15: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 2s 20ms/step - loss: 0.4492 - accuracy: 0.8349 - val_loss: 0.7702 - val_accuracy: 0.7850
  Epoch 16/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4024 - accuracy: 0.8490
  Epoch 16: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 2s 20ms/step - loss: 0.4057 - accuracy: 0.8479 - val_loss: 0.7375 - val_accuracy: 0.8014
  Epoch 17/40
  121/121 [==============================] - ETA: 0s - loss: 0.3776 - accuracy: 0.8589
  Epoch 17: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 22ms/step - loss: 0.3776 - accuracy: 0.8589 - val_loss: 0.7692 - val_accuracy: 0.7991
  Epoch 18/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3786 - accuracy: 0.8594
  Epoch 18: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 21ms/step - loss: 0.3786 - accuracy: 0.8586 - val_loss: 0.8283 - val_accuracy: 0.7874
  Epoch 19/40
  120/121 [============================>.] - ETA: 0s - loss: 0.3526 - accuracy: 0.8641
  Epoch 19: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 21ms/step - loss: 0.3523 - accuracy: 0.8643 - val_loss: 0.7889 - val_accuracy: 0.7944
  Epoch 20/40
  119/121 [============================>.] - ETA: 0s - loss: 0.3298 - accuracy: 0.8737
  Epoch 20: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 27ms/step - loss: 0.3314 - accuracy: 0.8734 - val_loss: 0.8751 - val_accuracy: 0.7804
  Epoch 21/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3306 - accuracy: 0.8718
  Epoch 21: val_accuracy improved from 0.80140 to 0.80841, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 0.3314 - accuracy: 0.8711 - val_loss: 0.7938 - val_accuracy: 0.8084
  Epoch 22/40
  121/121 [==============================] - ETA: 0s - loss: 0.3051 - accuracy: 0.8807
  Epoch 22: val_accuracy did not improve from 0.80841
  121/121 [==============================] - 2s 20ms/step - loss: 0.3051 - accuracy: 0.8807 - val_loss: 0.7822 - val_accuracy: 0.7921
  Epoch 22: early stopping
  

The training stopped at epoch 21 because the validation loss had increased during 10 epochs. The highest validation accuracy was 0.780 and validation loss 0.7628.

Model 2: Accuracy vs epochs plot

plt.plot(history2.history['accuracy'])
  plt.plot(history2.history['val_accuracy'])
  plt.title('model2 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

Notice that somewhere around epoch 15 the validation loss begins to increase.

Overall Model2 seems to perform better than the 2 previous models.

Evaluation of model2 model.evaluate using the Test data

# Evaluate the model on the test data
  score2 = model2.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score2[0])
  print('Test accuracy:', score2[1])
  

Test loss: 0.9905914068222046
  Test accuracy: 0.7873684167861938
  

# Test Prediction
  y_pred = model2.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 7ms/step
  

# Test Accuracy
  accuracyScore2 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore2)
  

0.7873684210526316
  

Classification Report

# Compute the classification report
  cr2 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr2)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.86      0.86      0.86        50
                    Fat Hen       0.75      0.88      0.81        48
            Shepherds Purse       0.78      0.30      0.44        23
               Common wheat       0.73      0.73      0.73        22
           Common Chickweed       0.92      0.95      0.94        61
                   Charlock       0.90      0.92      0.91        39
                   Cleavers       0.81      0.86      0.83        29
          Scentless Mayweed       0.70      0.75      0.72        52
                 Sugar beet       0.88      0.76      0.82        38
                      Maize       0.65      0.59      0.62        22
                Black-grass       0.50      0.15      0.24        26
           Loose Silky-bent       0.71      0.95      0.82        65

                   accuracy                           0.79       475
                  macro avg       0.76      0.73      0.73       475
               weighted avg       0.78      0.79      0.77       475

  

The overall accuracy of Model2 is 0.79, which is a ok. The precision and recall values for each class are also mostly good, with some classes having higher values than others.

The highest precision and recall values are achieved for "Small-flowered Cranesbill", "Fat Hen", "Common Chickweed", "Charlock", and "Cleavers", which means that the model can identify well these classes.

Some classes have lower precision and recall values, such as "Shepherds Purse", "Common wheat", "Scentless Mayweed", "Sugar beet", "Maize", and "Black-grass".

Scoresdf = Scoresdf.append({'model':'Model2','accuracy':accuracyScore2,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

#set and plot the Confusion matrix
  cm2 = confusion_matrix(normal_y_test, y_test_pred_classes)

  plt.figure(figsize=(8,6))
  sns.heatmap(cm2, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model2 performs better than Model0 and Model1. It actually performs OK.

But we can improve the model for better performance.

Model 3

To try to improve Model2, we change the optimizer from Adam to SGD.

• Layers : 10

• Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

• Layer 2: MaxPooling2D layer
• Layer 3: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 4: MaxPooling2D layer
• Layer 5: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu
• Layer 6: MaxPooling2D layer
• Layer 7: Flatten layer
• Layer 8: Dense layer : 128 neurons, Relu

• Layer 9: Droput layer: 0.5

• Optimizer: SGD

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Lets used SGD as optimizer with the second model since we obtained a higher accuracy than the first model

# Using SGD Optimizer
  sgd = SGD(learning_rate=0.01, momentum=0.9)
  

Model Implementation

# Define the model architecture
  model3 = Sequential()

  #Convolution with 32 filters size 3,3 and relu
  model3.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=(128,128,3)))

  #add max pooling to reduce the size of output of first convolution layer
  model3.add(MaxPooling2D(pool_size=(2,2)))

  #Convolution with 64 filters size 3,3 and relu
  model3.add(Conv2D(64, kernel_size=(3,3), activation='relu'))

  #add max pooling to reduce the size of output of first convolution layer
  model3.add(MaxPooling2D(pool_size=(2,2)))

  #Convolution with 128 filters size 3,3 and relu
  model3.add(Conv2D(128, kernel_size=(3,3), activation='relu'))

  #add max pooling to reduce the size of output of first convolution layer
  model3.add(MaxPooling2D(pool_size=(2,2)))

  #reshape the output to a flat vector
  model3.add(Flatten())

  #fully connected layer with 128 neurons
  model3.add(Dense(128, activation='relu'))

  #add dropout to prevent overfitting
  model3.add(Dropout(0.5))

  #add output layer
  model3.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model3.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

  # Print the model summary
  model3.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0
   2D)

   flatten (Flatten)           (None, 25088)             0

   dense (Dense)               (None, 128)               3211392

   dropout (Dropout)           (None, 128)               0

   dense_1 (Dense)             (None, 12)                1548

  =================================================================
  Total params: 3,306,188
  Trainable params: 3,306,188
  Non-trainable params: 0
  _________________________________________________________________
  

Model Visualization

#visualize the cnn model architecture
  plot_model(model3, to_file='model3_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize the cnn model architecture using visualkeras
  visualkeras.layered_view(model3, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 40 epochs and validation_split as 10%
  history3 = model3.fit(X_train, y_train_onehot, epochs=40, batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/40
  121/121 [==============================] - ETA: 0s - loss: 2.3845 - accuracy: 0.1578
  Epoch 1: val_accuracy improved from -inf to 0.32243, saving model to best_model.h5
  121/121 [==============================] - 4s 26ms/step - loss: 2.3845 - accuracy: 0.1578 - val_loss: 2.0716 - val_accuracy: 0.3224
  Epoch 2/40
  118/121 [============================>.] - ETA: 0s - loss: 1.8602 - accuracy: 0.3665
  Epoch 2: val_accuracy did not improve from 0.32243
  121/121 [==============================] - 2s 20ms/step - loss: 1.8570 - accuracy: 0.3678 - val_loss: 2.1210 - val_accuracy: 0.2967
  Epoch 3/40
  118/121 [============================>.] - ETA: 0s - loss: 1.6527 - accuracy: 0.4256
  Epoch 3: val_accuracy improved from 0.32243 to 0.45794, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 1.6484 - accuracy: 0.4273 - val_loss: 1.5749 - val_accuracy: 0.4579
  Epoch 4/40
  119/121 [============================>.] - ETA: 0s - loss: 1.4911 - accuracy: 0.4811
  Epoch 4: val_accuracy did not improve from 0.45794
  121/121 [==============================] - 2s 19ms/step - loss: 1.4852 - accuracy: 0.4830 - val_loss: 1.4450 - val_accuracy: 0.4579
  Epoch 5/40
  118/121 [============================>.] - ETA: 0s - loss: 1.3449 - accuracy: 0.5246
  Epoch 5: val_accuracy improved from 0.45794 to 0.46028, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 1.3446 - accuracy: 0.5251 - val_loss: 1.6715 - val_accuracy: 0.4603
  Epoch 6/40
  118/121 [============================>.] - ETA: 0s - loss: 1.2553 - accuracy: 0.5662
  Epoch 6: val_accuracy improved from 0.46028 to 0.50234, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 1.2546 - accuracy: 0.5667 - val_loss: 1.4218 - val_accuracy: 0.5023
  Epoch 7/40
  118/121 [============================>.] - ETA: 0s - loss: 1.1140 - accuracy: 0.6163
  Epoch 7: val_accuracy improved from 0.50234 to 0.62850, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 1.1108 - accuracy: 0.6168 - val_loss: 1.1198 - val_accuracy: 0.6285
  Epoch 8/40
  120/121 [============================>.] - ETA: 0s - loss: 1.0393 - accuracy: 0.6391
  Epoch 8: val_accuracy improved from 0.62850 to 0.63551, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 1.0389 - accuracy: 0.6389 - val_loss: 1.0225 - val_accuracy: 0.6355
  Epoch 9/40
  119/121 [============================>.] - ETA: 0s - loss: 0.9215 - accuracy: 0.6786
  Epoch 9: val_accuracy improved from 0.63551 to 0.69393, saving model to best_model.h5
  121/121 [==============================] - 2s 21ms/step - loss: 0.9207 - accuracy: 0.6792 - val_loss: 0.9386 - val_accuracy: 0.6939
  Epoch 10/40
  118/121 [============================>.] - ETA: 0s - loss: 0.9077 - accuracy: 0.6851
  Epoch 10: val_accuracy did not improve from 0.69393
  121/121 [==============================] - 2s 20ms/step - loss: 0.9055 - accuracy: 0.6862 - val_loss: 0.9669 - val_accuracy: 0.6799
  Epoch 11/40
  118/121 [============================>.] - ETA: 0s - loss: 0.7901 - accuracy: 0.7219
  Epoch 11: val_accuracy did not improve from 0.69393
  121/121 [==============================] - 2s 20ms/step - loss: 0.7900 - accuracy: 0.7219 - val_loss: 1.0805 - val_accuracy: 0.6355
  Epoch 12/40
  119/121 [============================>.] - ETA: 0s - loss: 0.7174 - accuracy: 0.7453
  Epoch 12: val_accuracy improved from 0.69393 to 0.73364, saving model to best_model.h5
  121/121 [==============================] - 3s 21ms/step - loss: 0.7178 - accuracy: 0.7447 - val_loss: 0.8993 - val_accuracy: 0.7336
  Epoch 13/40
  119/121 [============================>.] - ETA: 0s - loss: 0.6675 - accuracy: 0.7655
  Epoch 13: val_accuracy improved from 0.73364 to 0.73832, saving model to best_model.h5
  121/121 [==============================] - 3s 22ms/step - loss: 0.6708 - accuracy: 0.7640 - val_loss: 0.8316 - val_accuracy: 0.7383
  Epoch 14/40
  119/121 [============================>.] - ETA: 0s - loss: 0.6881 - accuracy: 0.7595
  Epoch 14: val_accuracy did not improve from 0.73832
  121/121 [==============================] - 2s 20ms/step - loss: 0.6882 - accuracy: 0.7596 - val_loss: 0.8765 - val_accuracy: 0.7383
  Epoch 15/40
  118/121 [============================>.] - ETA: 0s - loss: 0.6067 - accuracy: 0.7844
  Epoch 15: val_accuracy improved from 0.73832 to 0.79673, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 0.6055 - accuracy: 0.7848 - val_loss: 0.7043 - val_accuracy: 0.7967
  Epoch 16/40
  118/121 [============================>.] - ETA: 0s - loss: 0.5655 - accuracy: 0.7969
  Epoch 16: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 2s 20ms/step - loss: 0.5661 - accuracy: 0.7962 - val_loss: 0.8609 - val_accuracy: 0.7500
  Epoch 17/40
  119/121 [============================>.] - ETA: 0s - loss: 0.4954 - accuracy: 0.8180
  Epoch 17: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 2s 20ms/step - loss: 0.4939 - accuracy: 0.8186 - val_loss: 0.7640 - val_accuracy: 0.7593
  Epoch 18/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4738 - accuracy: 0.8244
  Epoch 18: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 3s 21ms/step - loss: 0.4720 - accuracy: 0.8251 - val_loss: 0.8498 - val_accuracy: 0.7734
  Epoch 19/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4325 - accuracy: 0.8392
  Epoch 19: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 3s 21ms/step - loss: 0.4319 - accuracy: 0.8391 - val_loss: 0.7769 - val_accuracy: 0.7710
  Epoch 20/40
  118/121 [============================>.] - ETA: 0s - loss: 0.4054 - accuracy: 0.8520
  Epoch 20: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 2s 20ms/step - loss: 0.4040 - accuracy: 0.8531 - val_loss: 0.8047 - val_accuracy: 0.7780
  Epoch 21/40
  119/121 [============================>.] - ETA: 0s - loss: 0.3600 - accuracy: 0.8640
  Epoch 21: val_accuracy improved from 0.79673 to 0.80140, saving model to best_model.h5
  121/121 [==============================] - 2s 20ms/step - loss: 0.3609 - accuracy: 0.8648 - val_loss: 0.8184 - val_accuracy: 0.8014
  Epoch 22/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3771 - accuracy: 0.8604
  Epoch 22: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 2s 19ms/step - loss: 0.3747 - accuracy: 0.8612 - val_loss: 0.8573 - val_accuracy: 0.7477
  Epoch 23/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3378 - accuracy: 0.8816
  Epoch 23: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 2s 20ms/step - loss: 0.3357 - accuracy: 0.8825 - val_loss: 0.8310 - val_accuracy: 0.7640
  Epoch 24/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3023 - accuracy: 0.8922
  Epoch 24: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 21ms/step - loss: 0.2997 - accuracy: 0.8932 - val_loss: 0.9448 - val_accuracy: 0.7593
  Epoch 25/40
  118/121 [============================>.] - ETA: 0s - loss: 0.3030 - accuracy: 0.8856
  Epoch 25: val_accuracy did not improve from 0.80140
  121/121 [==============================] - 3s 21ms/step - loss: 0.3038 - accuracy: 0.8856 - val_loss: 0.9751 - val_accuracy: 0.7827
  Epoch 25: early stopping
  

Training stopped on epoch 28. The highest validation accuracy was 0.7921 where the validation loss was 0.844.

Model3 Accuracy vs Epoch curve

plt.plot(history3.history['accuracy'])
  plt.plot(history3.history['val_accuracy'])
  plt.title('model3 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

Around epoch 20, the validation accuracy is highest. The model seems to be performing decently around epoch 22. There is some overfitting in the data.

Evaluation of model3 model.evaluate using the Test data

# Evaluate the model on the test data
  score3 = model3.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score3[0])
  print('Test accuracy:', score3[1])
  

Test loss: 0.9461641907691956
  Test accuracy: 0.7726315855979919
  

# Test Prediction
  y_pred = model3.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 7ms/step
  

# Test Accuracy
  accuracyScore3 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore3)
  

0.7726315789473684
  

Classification Report

# Compute the classification report
  cr3 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr3)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.83      0.88      0.85        50
                    Fat Hen       0.83      0.81      0.82        48
            Shepherds Purse       0.64      0.61      0.62        23
               Common wheat       0.83      0.45      0.59        22
           Common Chickweed       0.86      0.97      0.91        61
                   Charlock       0.86      0.92      0.89        39
                   Cleavers       0.91      0.72      0.81        29
          Scentless Mayweed       0.75      0.79      0.77        52
                 Sugar beet       0.72      0.74      0.73        38
                      Maize       0.65      0.68      0.67        22
                Black-grass       0.35      0.23      0.28        26
           Loose Silky-bent       0.74      0.83      0.78        65

                   accuracy                           0.77       475
                  macro avg       0.75      0.72      0.73       475
               weighted avg       0.77      0.77      0.76       475

  

Model3s overall accuracy is at 0.76, and the macro average F1-score is at 0.73, while the weighted average is at 0.7.

The model performs well on some classes, such as Small-flowered Cranesbill, Fat Hen, Common Chickweed, and Charlock, but doesn't perform well with others like Black-grass and Shepherds Purse.

Scoresdf = Scoresdf.append({'model':'Model3','accuracy':accuracyScore3,'layers':'10','total parameters':'3,306,188', 'neurons':'128', 'optimizer':'SGD'},ignore_index=True)
  

Confusion Matrix

# Computer the Confusion matrix
  cm3 = confusion_matrix(normal_y_test, y_test_pred_classes)

  plt.figure(figsize=(8,6))
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  sns.heatmap(cm3, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  

<Axes: >

Overall, model2 performs better than model3.

Model 4

Lets try adding more layers to try to improve the previous models.

Layers : 17

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: BatchNormalization()

Layer 3: MaxPooling2D layer

Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu

Layer 5: BatchNormalization()

Layer 6: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 10: Flatten layer

Layer 11: Dense layer : 256 neurons, Relu

Layer 12: BatchNormalization

Layer 13: Droput layer: 0.5

Optimizer: Adam

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# initialized a sequential model
  model4 = Sequential()

  model4.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
  model4.add(BatchNormalization())
  model4.add(MaxPooling2D(pool_size=(2,2)))

  model4.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model4.add(BatchNormalization())
  model4.add(MaxPooling2D(pool_size=(2,2)))

  model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model4.add(BatchNormalization())
  model4.add(MaxPooling2D(pool_size=(2,2)))

  model4.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model4.add(BatchNormalization())
  model4.add(MaxPooling2D(pool_size=(2,2)))

  model4.add(Flatten())
  model4.add(Dense(128, activation='relu'))
  model4.add(BatchNormalization())

  model4.add(Dropout(0.5))
  model4.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model4.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   batch_normalization (BatchN  (None, 126, 126, 32)     128
   ormalization)

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496

   batch_normalization_1 (Batc  (None, 61, 61, 64)       256
   hNormalization)

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856

   batch_normalization_2 (Batc  (None, 28, 28, 128)      512
   hNormalization)

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0
   2D)

   conv2d_3 (Conv2D)           (None, 12, 12, 128)       147584

   batch_normalization_3 (Batc  (None, 12, 12, 128)      512
   hNormalization)

   max_pooling2d_3 (MaxPooling  (None, 6, 6, 128)        0
   2D)

   flatten (Flatten)           (None, 4608)              0

   dense (Dense)               (None, 128)               589952

   batch_normalization_4 (Batc  (None, 128)              512
   hNormalization)

   dropout (Dropout)           (None, 128)               0

   dense_1 (Dense)             (None, 12)                1548

  =================================================================
  Total params: 834,252
  Trainable params: 833,292
  Non-trainable params: 960
  _________________________________________________________________
  

Model Visualization

#visualize model4 architecture
  plot_model(model4, to_file='model4_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize model4 architecture using visualkeras
  visualkeras.layered_view(model4, legend=True)
  

Early Stopping

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 100 epochs and validation_split as 10%
  history4=model4.fit(X_train,
            y_train_onehot,
            epochs=100,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/100
  121/121 [==============================] - ETA: 0s - loss: 1.6201 - accuracy: 0.5134
  Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 9s 31ms/step - loss: 1.6201 - accuracy: 0.5134 - val_loss: 5.6225 - val_accuracy: 0.1215
  Epoch 2/100
  120/121 [============================>.] - ETA: 0s - loss: 0.8899 - accuracy: 0.7138
  Epoch 2: val_accuracy did not improve from 0.12150
  121/121 [==============================] - 3s 27ms/step - loss: 0.8915 - accuracy: 0.7135 - val_loss: 10.7526 - val_accuracy: 0.1215
  Epoch 3/100
  119/121 [============================>.] - ETA: 0s - loss: 0.6252 - accuracy: 0.7915
  Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5
  121/121 [==============================] - 3s 28ms/step - loss: 0.6285 - accuracy: 0.7902 - val_loss: 10.3552 - val_accuracy: 0.1238
  Epoch 4/100
  119/121 [============================>.] - ETA: 0s - loss: 0.5196 - accuracy: 0.8267
  Epoch 4: val_accuracy improved from 0.12383 to 0.24299, saving model to best_model.h5
  121/121 [==============================] - 4s 32ms/step - loss: 0.5195 - accuracy: 0.8269 - val_loss: 4.9083 - val_accuracy: 0.2430
  Epoch 5/100
  120/121 [============================>.] - ETA: 0s - loss: 0.4166 - accuracy: 0.8544
  Epoch 5: val_accuracy improved from 0.24299 to 0.30374, saving model to best_model.h5
  121/121 [==============================] - 3s 28ms/step - loss: 0.4166 - accuracy: 0.8547 - val_loss: 3.4493 - val_accuracy: 0.3037
  Epoch 6/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3413 - accuracy: 0.8857
  Epoch 6: val_accuracy improved from 0.30374 to 0.59579, saving model to best_model.h5
  121/121 [==============================] - 3s 28ms/step - loss: 0.3429 - accuracy: 0.8854 - val_loss: 1.3685 - val_accuracy: 0.5958
  Epoch 7/100
  119/121 [============================>.] - ETA: 0s - loss: 0.2859 - accuracy: 0.9097
  Epoch 7: val_accuracy improved from 0.59579 to 0.86682, saving model to best_model.h5
  121/121 [==============================] - 3s 27ms/step - loss: 0.2853 - accuracy: 0.9098 - val_loss: 0.3992 - val_accuracy: 0.8668
  Epoch 8/100
  119/121 [============================>.] - ETA: 0s - loss: 0.2253 - accuracy: 0.9315
  Epoch 8: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 4s 30ms/step - loss: 0.2259 - accuracy: 0.9306 - val_loss: 1.2766 - val_accuracy: 0.6332
  Epoch 9/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1847 - accuracy: 0.9459
  Epoch 9: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 3s 28ms/step - loss: 0.1850 - accuracy: 0.9459 - val_loss: 0.4840 - val_accuracy: 0.8248
  Epoch 10/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1638 - accuracy: 0.9474
  Epoch 10: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 3s 27ms/step - loss: 0.1647 - accuracy: 0.9470 - val_loss: 1.0341 - val_accuracy: 0.7009
  Epoch 11/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1588 - accuracy: 0.9451
  Epoch 11: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 3s 27ms/step - loss: 0.1594 - accuracy: 0.9449 - val_loss: 1.2241 - val_accuracy: 0.6799
  Epoch 12/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1690 - accuracy: 0.9451
  Epoch 12: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 3s 28ms/step - loss: 0.1699 - accuracy: 0.9444 - val_loss: 1.9335 - val_accuracy: 0.5818
  Epoch 13/100
  121/121 [==============================] - ETA: 0s - loss: 0.1456 - accuracy: 0.9485
  Epoch 13: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 3s 29ms/step - loss: 0.1456 - accuracy: 0.9485 - val_loss: 0.7255 - val_accuracy: 0.7687
  Epoch 14/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1349 - accuracy: 0.9568
  Epoch 14: val_accuracy improved from 0.86682 to 0.86916, saving model to best_model.h5
  121/121 [==============================] - 3s 27ms/step - loss: 0.1356 - accuracy: 0.9563 - val_loss: 0.4801 - val_accuracy: 0.8692
  Epoch 15/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1090 - accuracy: 0.9648
  Epoch 15: val_accuracy did not improve from 0.86916
  121/121 [==============================] - 3s 27ms/step - loss: 0.1103 - accuracy: 0.9646 - val_loss: 0.5455 - val_accuracy: 0.8178
  Epoch 16/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0723 - accuracy: 0.9812
  Epoch 16: val_accuracy did not improve from 0.86916
  121/121 [==============================] - 3s 27ms/step - loss: 0.0723 - accuracy: 0.9813 - val_loss: 0.6476 - val_accuracy: 0.7944
  Epoch 17/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9819
  Epoch 17: val_accuracy did not improve from 0.86916
  121/121 [==============================] - 3s 29ms/step - loss: 0.0629 - accuracy: 0.9821 - val_loss: 1.0993 - val_accuracy: 0.7243
  Epoch 17: early stopping
  

Training stopped on epoch 33. The highest validation accuracy was 0.8925 where the validation loss was 0.3573.

Model4 Accuracy vs Epoch curve

plt.plot(history4.history['accuracy'])
  plt.plot(history4.history['val_accuracy'])
  plt.title('model4 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

Notice that the model tends to overfit as epochs increased.

Evaluation of model4 model.evaluate using the Test data

# Evaluate the model on the test data
  score4 = model4.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score4[0])
  print('Test accuracy:', score4[1])
  

Test loss: 1.3278809785842896
  Test accuracy: 0.7052631378173828
  

# Test Prediction
  y_pred = model4.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 8ms/step
  

# Test Accuracy
  accuracyScore4 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore4)
  

0.7052631578947368
  

Classification Report

# Compute the classification report
  cr4 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr4)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.95      0.78      0.86        50
                    Fat Hen       0.47      0.98      0.64        48
            Shepherds Purse       0.53      0.43      0.48        23
               Common wheat       0.82      0.41      0.55        22
           Common Chickweed       0.91      0.82      0.86        61
                   Charlock       1.00      0.82      0.90        39
                   Cleavers       0.96      0.86      0.91        29
          Scentless Mayweed       1.00      0.33      0.49        52
                 Sugar beet       0.85      0.76      0.81        38
                      Maize       0.51      0.95      0.67        22
                Black-grass       0.25      0.27      0.26        26
           Loose Silky-bent       0.68      0.75      0.72        65

                   accuracy                           0.71       475
                  macro avg       0.74      0.68      0.68       475
               weighted avg       0.78      0.71      0.70       475

  

Model4 has an overall accuracy of 83%.

The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.

The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.

The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.

Scoresdf = Scoresdf.append({'model':'Model4','accuracy':accuracyScore4,'layers':'17','total parameters':'834,252', 'neurons':'128', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Compute Confusion matrix
  cm4 = confusion_matrix(normal_y_test, y_test_pred_classes)
  plt.figure(figsize=(8,6))
  sns.heatmap(cm4, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model4 seems to perform well overall, but there is some variation in the performance of the model for different classes.

Model 5

To continue trying to improve the previous model, we add more layers to our model.

Layers : 19

Layer 1: Convolutional layer : 32 filters, Kernel_size = (3,3), Relu

Layer 2: BatchNormalization()

Layer 3: MaxPooling2D layer

Layer 4: Convolutional layer : 64 filters, Kernel_size = (3,3), Relu

Layer 5: BatchNormalization()

Layer 6: MaxPooling2D layer

Layer 7: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 8: BatchNormalization()

Layer 9: MaxPooling2D layer

Layer 10: Convolutional layer : 256 filters, Kernel_size = (3,3), Relu

Layer 11: BatchNormalization()

Layer 12: MaxPooling2D layer

Layer 13: Convolutional layer : 128 filters, Kernel_size = (3,3), Relu

Layer 14: BatchNormalization()

Layer 15: MaxPooling2D layer

Layer 16: Flatten layer

Layer 17: Dense layer : 256 neurons, Relu

Layer 18: BatchNormalization

Layer 19: Droput layer: 0.5

Optimizer: Adam

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# initialized a sequential model
  model5 = Sequential()

  model5.add(Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape))
  model5.add(BatchNormalization())
  model5.add(MaxPooling2D(pool_size=(2,2)))

  model5.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model5.add(BatchNormalization())
  model5.add(MaxPooling2D(pool_size=(2,2)))

  model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model5.add(BatchNormalization())
  model5.add(MaxPooling2D(pool_size=(2,2)))

  model5.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
  model5.add(BatchNormalization())
  model5.add(MaxPooling2D(pool_size=(2,2)))

  model5.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model5.add(BatchNormalization())
  model5.add(MaxPooling2D(pool_size=(2,2)))

  model5.add(Flatten())
  model5.add(Dense(256, activation='relu'))
  model5.add(BatchNormalization())
  model5.add(Dropout(0.5))

  model5.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model5.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model5.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   batch_normalization (BatchN  (None, 126, 126, 32)     128
   ormalization)

   max_pooling2d (MaxPooling2D  (None, 63, 63, 32)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 64)        18496

   batch_normalization_1 (Batc  (None, 61, 61, 64)       256
   hNormalization)

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 64)       0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 128)       73856

   batch_normalization_2 (Batc  (None, 28, 28, 128)      512
   hNormalization)

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 128)      0
   2D)

   conv2d_3 (Conv2D)           (None, 12, 12, 256)       295168

   batch_normalization_3 (Batc  (None, 12, 12, 256)      1024
   hNormalization)

   max_pooling2d_3 (MaxPooling  (None, 6, 6, 256)        0
   2D)

   conv2d_4 (Conv2D)           (None, 4, 4, 128)         295040

   batch_normalization_4 (Batc  (None, 4, 4, 128)        512
   hNormalization)

   max_pooling2d_4 (MaxPooling  (None, 2, 2, 128)        0
   2D)

   flatten (Flatten)           (None, 512)               0

   dense (Dense)               (None, 256)               131328

   batch_normalization_5 (Batc  (None, 256)              1024
   hNormalization)

   dropout (Dropout)           (None, 256)               0

   dense_1 (Dense)             (None, 12)                3084

  =================================================================
  Total params: 821,324
  Trainable params: 819,596
  Non-trainable params: 1,728
  _________________________________________________________________
  

Model5 Architecture Visualization

##visualize model architecture
  plot_model(model5, to_file='model5_plot.png', show_shapes=True, show_layer_names=True)
  

##model architecture visualization using visualkeras
  visualkeras.layered_view(model5, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=5)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 100 epochs and validation_split as 10%
  history5 = model5.fit(X_train,
            y_train_onehot,
            epochs=100,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/100
  119/121 [============================>.] - ETA: 0s - loss: 1.6716 - accuracy: 0.5011
  Epoch 1: val_accuracy improved from -inf to 0.06308, saving model to best_model.h5
  121/121 [==============================] - 9s 36ms/step - loss: 1.6664 - accuracy: 0.5032 - val_loss: 7.5329 - val_accuracy: 0.0631
  Epoch 2/100
  119/121 [============================>.] - ETA: 0s - loss: 0.8780 - accuracy: 0.7124
  Epoch 2: val_accuracy improved from 0.06308 to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 4s 32ms/step - loss: 0.8768 - accuracy: 0.7130 - val_loss: 13.4861 - val_accuracy: 0.1215
  Epoch 3/100
  119/121 [============================>.] - ETA: 0s - loss: 0.6572 - accuracy: 0.7723
  Epoch 3: val_accuracy improved from 0.12150 to 0.14486, saving model to best_model.h5
  121/121 [==============================] - 4s 30ms/step - loss: 0.6564 - accuracy: 0.7723 - val_loss: 12.1352 - val_accuracy: 0.1449
  Epoch 4/100
  120/121 [============================>.] - ETA: 0s - loss: 0.4920 - accuracy: 0.8268
  Epoch 4: val_accuracy improved from 0.14486 to 0.15888, saving model to best_model.h5
  121/121 [==============================] - 4s 29ms/step - loss: 0.4933 - accuracy: 0.8264 - val_loss: 7.2529 - val_accuracy: 0.1589
  Epoch 5/100
  119/121 [============================>.] - ETA: 0s - loss: 0.4023 - accuracy: 0.8595
  Epoch 5: val_accuracy improved from 0.15888 to 0.54907, saving model to best_model.h5
  121/121 [==============================] - 4s 32ms/step - loss: 0.4035 - accuracy: 0.8594 - val_loss: 1.6262 - val_accuracy: 0.5491
  Epoch 6/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3601 - accuracy: 0.8763
  Epoch 6: val_accuracy improved from 0.54907 to 0.72664, saving model to best_model.h5
  121/121 [==============================] - 4s 31ms/step - loss: 0.3619 - accuracy: 0.8760 - val_loss: 0.8660 - val_accuracy: 0.7266
  Epoch 7/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2981 - accuracy: 0.8898
  Epoch 7: val_accuracy did not improve from 0.72664
  121/121 [==============================] - 3s 28ms/step - loss: 0.2979 - accuracy: 0.8900 - val_loss: 0.9416 - val_accuracy: 0.6963
  Epoch 8/100
  119/121 [============================>.] - ETA: 0s - loss: 0.2452 - accuracy: 0.9136
  Epoch 8: val_accuracy improved from 0.72664 to 0.78271, saving model to best_model.h5
  121/121 [==============================] - 3s 29ms/step - loss: 0.2483 - accuracy: 0.9127 - val_loss: 0.6978 - val_accuracy: 0.7827
  Epoch 9/100
  119/121 [============================>.] - ETA: 0s - loss: 0.2397 - accuracy: 0.9107
  Epoch 9: val_accuracy did not improve from 0.78271
  121/121 [==============================] - 4s 30ms/step - loss: 0.2394 - accuracy: 0.9108 - val_loss: 1.1052 - val_accuracy: 0.6776
  Epoch 10/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1977 - accuracy: 0.9244
  Epoch 10: val_accuracy improved from 0.78271 to 0.81542, saving model to best_model.h5
  121/121 [==============================] - 4s 33ms/step - loss: 0.1987 - accuracy: 0.9236 - val_loss: 0.5499 - val_accuracy: 0.8154
  Epoch 11/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1850 - accuracy: 0.9288
  Epoch 11: val_accuracy improved from 0.81542 to 0.83879, saving model to best_model.h5
  121/121 [==============================] - 4s 29ms/step - loss: 0.1846 - accuracy: 0.9288 - val_loss: 0.5094 - val_accuracy: 0.8388
  Epoch 12/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1423 - accuracy: 0.9533
  Epoch 12: val_accuracy improved from 0.83879 to 0.84112, saving model to best_model.h5
  121/121 [==============================] - 3s 29ms/step - loss: 0.1429 - accuracy: 0.9530 - val_loss: 0.5343 - val_accuracy: 0.8411
  Epoch 13/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1648 - accuracy: 0.9406
  Epoch 13: val_accuracy did not improve from 0.84112
  121/121 [==============================] - 4s 30ms/step - loss: 0.1648 - accuracy: 0.9407 - val_loss: 0.7414 - val_accuracy: 0.7804
  Epoch 14/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1324 - accuracy: 0.9554
  Epoch 14: val_accuracy did not improve from 0.84112
  121/121 [==============================] - 4s 31ms/step - loss: 0.1320 - accuracy: 0.9558 - val_loss: 0.7250 - val_accuracy: 0.8107
  Epoch 15/100
  119/121 [============================>.] - ETA: 0s - loss: 0.1177 - accuracy: 0.9624
  Epoch 15: val_accuracy did not improve from 0.84112
  121/121 [==============================] - 3s 28ms/step - loss: 0.1198 - accuracy: 0.9618 - val_loss: 0.7258 - val_accuracy: 0.7944
  Epoch 16/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1284 - accuracy: 0.9544
  Epoch 16: val_accuracy did not improve from 0.84112
  121/121 [==============================] - 3s 28ms/step - loss: 0.1285 - accuracy: 0.9543 - val_loss: 0.4891 - val_accuracy: 0.8411
  Epoch 17/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1123 - accuracy: 0.9620
  Epoch 17: val_accuracy did not improve from 0.84112
  121/121 [==============================] - 3s 28ms/step - loss: 0.1122 - accuracy: 0.9620 - val_loss: 0.6232 - val_accuracy: 0.8014
  Epoch 18/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0725 - accuracy: 0.9764
  Epoch 18: val_accuracy improved from 0.84112 to 0.85748, saving model to best_model.h5
  121/121 [==============================] - 4s 32ms/step - loss: 0.0727 - accuracy: 0.9763 - val_loss: 0.4822 - val_accuracy: 0.8575
  Epoch 19/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0758 - accuracy: 0.9751
  Epoch 19: val_accuracy did not improve from 0.85748
  121/121 [==============================] - 4s 30ms/step - loss: 0.0755 - accuracy: 0.9750 - val_loss: 0.4922 - val_accuracy: 0.8458
  Epoch 20/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0633 - accuracy: 0.9794
  Epoch 20: val_accuracy did not improve from 0.85748
  121/121 [==============================] - 3s 28ms/step - loss: 0.0646 - accuracy: 0.9789 - val_loss: 1.2289 - val_accuracy: 0.6822
  Epoch 21/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1150 - accuracy: 0.9578
  Epoch 21: val_accuracy did not improve from 0.85748
  121/121 [==============================] - 3s 28ms/step - loss: 0.1156 - accuracy: 0.9576 - val_loss: 0.6274 - val_accuracy: 0.8201
  Epoch 22/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0844 - accuracy: 0.9688
  Epoch 22: val_accuracy did not improve from 0.85748
  121/121 [==============================] - 4s 30ms/step - loss: 0.0837 - accuracy: 0.9691 - val_loss: 1.7372 - val_accuracy: 0.6986
  Epoch 23/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9863
  Epoch 23: val_accuracy improved from 0.85748 to 0.87617, saving model to best_model.h5
  121/121 [==============================] - 4s 30ms/step - loss: 0.0454 - accuracy: 0.9865 - val_loss: 0.4484 - val_accuracy: 0.8762
  Epoch 24/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0505 - accuracy: 0.9854
  Epoch 24: val_accuracy improved from 0.87617 to 0.87850, saving model to best_model.h5
  121/121 [==============================] - 4s 29ms/step - loss: 0.0504 - accuracy: 0.9854 - val_loss: 0.3818 - val_accuracy: 0.8785
  Epoch 25/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0419 - accuracy: 0.9878
  Epoch 25: val_accuracy did not improve from 0.87850
  121/121 [==============================] - 3s 28ms/step - loss: 0.0423 - accuracy: 0.9875 - val_loss: 0.7614 - val_accuracy: 0.8248
  Epoch 26/100
  119/121 [============================>.] - ETA: 0s - loss: 0.0427 - accuracy: 0.9856
  Epoch 26: val_accuracy did not improve from 0.87850
  121/121 [==============================] - 4s 31ms/step - loss: 0.0425 - accuracy: 0.9854 - val_loss: 0.8043 - val_accuracy: 0.8271
  Epoch 27/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0470 - accuracy: 0.9844
  Epoch 27: val_accuracy did not improve from 0.87850
  121/121 [==============================] - 4s 31ms/step - loss: 0.0473 - accuracy: 0.9841 - val_loss: 3.3697 - val_accuracy: 0.5093
  Epoch 28/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0518 - accuracy: 0.9844
  Epoch 28: val_accuracy did not improve from 0.87850
  121/121 [==============================] - 3s 28ms/step - loss: 0.0518 - accuracy: 0.9844 - val_loss: 0.5574 - val_accuracy: 0.8435
  Epoch 29/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0408 - accuracy: 0.9875
  Epoch 29: val_accuracy did not improve from 0.87850
  121/121 [==============================] - 3s 28ms/step - loss: 0.0407 - accuracy: 0.9875 - val_loss: 1.0695 - val_accuracy: 0.7734
  Epoch 29: early stopping
  

Training stopped at epoch 15. The highest validation accuracy was 0.8575 and validation loss is 0.4383.

Accuracy vs Epoch curve

plt.plot(history5.history['accuracy'])
  plt.plot(history5.history['val_accuracy'])
  plt.title('model5 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

The model is overfitting. However, around epoch 9, it has a high validation accuracy and a high training accuracy.

Evaluation of model5 model.evaluate using the Test data

# Evaluate the model on the test data
  score5 = model5.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score5[0])
  print('Test accuracy:', score5[1])
  

Test loss: 1.1474685668945312
  Test accuracy: 0.7557894587516785
  

# Test Prediction
  y_pred = model5.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 11ms/step
  

# Test Accuracy
  accuracyScore5 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore5)
  

0.7557894736842106
  

Classification Report

# Compute the classification report
  cr5 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr5)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.94      0.92      0.93        50
                    Fat Hen       0.95      0.85      0.90        48
            Shepherds Purse       0.81      0.74      0.77        23
               Common wheat       0.82      0.64      0.72        22
           Common Chickweed       0.98      0.82      0.89        61
                   Charlock       0.97      0.90      0.93        39
                   Cleavers       0.93      0.86      0.89        29
          Scentless Mayweed       0.44      0.92      0.60        52
                 Sugar beet       1.00      0.21      0.35        38
                      Maize       0.53      0.77      0.63        22
                Black-grass       0.64      0.27      0.38        26
           Loose Silky-bent       0.71      0.78      0.74        65

                   accuracy                           0.76       475
                  macro avg       0.81      0.72      0.73       475
               weighted avg       0.82      0.76      0.75       475

  

The accuracy of the model5 is 83%, and the weighted average F1-score is 0.82. The model performs relatively well in predicting most of the plant species, with high precision, recall and f1-scores, except for Black-grass.

However Model4 seems to be performing slightly better.

Scoresdf = Scoresdf.append({'model':'Model5','accuracy':accuracyScore5,'layers':'20','total parameters':'821,324', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

#Visualize the Confusion matrix
  cm5 = confusion_matrix(normal_y_test, y_test_pred_classes)
  plt.figure(figsize=(8,6))

  sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model5 performs well for most classes but Black-grass.

Model 6

To try to improve the performance of our previous models, we changethe layers and filters.

Layers: 16

• Layer 1: Convolution, filter: 64, kernel 3,3
• Layer 2: BatchNormalization
• Layer 3: MaxPooling2D
• Layer 4: Convolution, filter 128, kernel 3,3
• Layer 5: BatchNormalization
• Layer 6: MaxPooling2D
• Layer 7: Convolution, filter 256, kernel 3,3
• Layer 8: BatchNormalization
• Layer 9: MaxPooling2D
• Layer 10: Convolution, filter 512, kernel 3,3
• Layer 11: BatchNormalization
• Layer 12: MaxPooling2D
• Layer 13: Flatten
• Layer 14: Dense 256
• Layer 15: BatchNormalization
• Layer 16: Dropout 0.5

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# initialized a sequential model
  model6 = Sequential()

  model6.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
  model6.add(BatchNormalization())
  model6.add(MaxPooling2D(pool_size=(2,2)))

  model6.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model6.add(BatchNormalization())
  model6.add(MaxPooling2D(pool_size=(2,2)))

  model6.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
  model6.add(BatchNormalization())
  model6.add(MaxPooling2D(pool_size=(2,2)))

  model6.add(Flatten())
  model6.add(Dense(512, activation='relu'))
  model6.add(BatchNormalization())
  model6.add(Dropout(0.5))

  model6.add(Dense(256, activation='relu'))
  model6.add(BatchNormalization())
  model6.add(Dropout(0.5))

  model6.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model6.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model6.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 64)      1792

   batch_normalization (BatchN  (None, 126, 126, 64)     256
   ormalization)

   max_pooling2d (MaxPooling2D  (None, 63, 63, 64)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 128)       73856

   batch_normalization_1 (Batc  (None, 61, 61, 128)      512
   hNormalization)

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 128)      0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 256)       295168

   batch_normalization_2 (Batc  (None, 28, 28, 256)      1024
   hNormalization)

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 256)      0
   2D)

   flatten (Flatten)           (None, 50176)             0

   dense (Dense)               (None, 512)               25690624

   batch_normalization_3 (Batc  (None, 512)              2048
   hNormalization)

   dropout (Dropout)           (None, 512)               0

   dense_1 (Dense)             (None, 256)               131328

   batch_normalization_4 (Batc  (None, 256)              1024
   hNormalization)

   dropout_1 (Dropout)         (None, 256)               0

   dense_2 (Dense)             (None, 12)                3084

  =================================================================
  Total params: 26,200,716
  Trainable params: 26,198,284
  Non-trainable params: 2,432
  _________________________________________________________________
  

Model6 Architecture Visualization

#visualize model6 architecture
  plot_model(model6, to_file='model6_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize model6 architecture using visualkeras
  visualkeras.layered_view(model6, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 40 epochs and validation_split as 10%
  history6=model6.fit(X_train,
            y_train_onehot,
            epochs=40,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/40
  120/121 [============================>.] - ETA: 0s - loss: 1.9669 - accuracy: 0.4292
  Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 18s 98ms/step - loss: 1.9666 - accuracy: 0.4292 - val_loss: 7.0356 - val_accuracy: 0.1215
  Epoch 2/40
  120/121 [============================>.] - ETA: 0s - loss: 1.1810 - accuracy: 0.6206
  Epoch 2: val_accuracy did not improve from 0.12150
  121/121 [==============================] - 6s 53ms/step - loss: 1.1832 - accuracy: 0.6200 - val_loss: 8.4802 - val_accuracy: 0.1215
  Epoch 3/40
  120/121 [============================>.] - ETA: 0s - loss: 0.8575 - accuracy: 0.7164
  Epoch 3: val_accuracy improved from 0.12150 to 0.12383, saving model to best_model.h5
  121/121 [==============================] - 12s 99ms/step - loss: 0.8580 - accuracy: 0.7161 - val_loss: 6.8681 - val_accuracy: 0.1238
  Epoch 4/40
  121/121 [==============================] - ETA: 0s - loss: 0.6646 - accuracy: 0.7720
  Epoch 4: val_accuracy improved from 0.12383 to 0.20794, saving model to best_model.h5
  121/121 [==============================] - 13s 109ms/step - loss: 0.6646 - accuracy: 0.7720 - val_loss: 5.8048 - val_accuracy: 0.2079
  Epoch 5/40
  121/121 [==============================] - ETA: 0s - loss: 0.5248 - accuracy: 0.8131
  Epoch 5: val_accuracy improved from 0.20794 to 0.42290, saving model to best_model.h5
  121/121 [==============================] - 12s 102ms/step - loss: 0.5248 - accuracy: 0.8131 - val_loss: 2.2491 - val_accuracy: 0.4229
  Epoch 6/40
  121/121 [==============================] - ETA: 0s - loss: 0.4506 - accuracy: 0.8451
  Epoch 6: val_accuracy improved from 0.42290 to 0.58879, saving model to best_model.h5
  121/121 [==============================] - 12s 100ms/step - loss: 0.4506 - accuracy: 0.8451 - val_loss: 1.3419 - val_accuracy: 0.5888
  Epoch 7/40
  121/121 [==============================] - ETA: 0s - loss: 0.3584 - accuracy: 0.8713
  Epoch 7: val_accuracy improved from 0.58879 to 0.62150, saving model to best_model.h5
  121/121 [==============================] - 12s 101ms/step - loss: 0.3584 - accuracy: 0.8713 - val_loss: 1.1451 - val_accuracy: 0.6215
  Epoch 8/40
  120/121 [============================>.] - ETA: 0s - loss: 0.3424 - accuracy: 0.8823
  Epoch 8: val_accuracy improved from 0.62150 to 0.76636, saving model to best_model.h5
  121/121 [==============================] - 11s 94ms/step - loss: 0.3438 - accuracy: 0.8815 - val_loss: 0.7611 - val_accuracy: 0.7664
  Epoch 9/40
  121/121 [==============================] - ETA: 0s - loss: 0.3091 - accuracy: 0.8895
  Epoch 9: val_accuracy did not improve from 0.76636
  121/121 [==============================] - 6s 52ms/step - loss: 0.3091 - accuracy: 0.8895 - val_loss: 2.2853 - val_accuracy: 0.3715
  Epoch 10/40
  120/121 [============================>.] - ETA: 0s - loss: 0.3499 - accuracy: 0.8802
  Epoch 10: val_accuracy did not improve from 0.76636
  121/121 [==============================] - 6s 52ms/step - loss: 0.3516 - accuracy: 0.8799 - val_loss: 6.1112 - val_accuracy: 0.1706
  Epoch 11/40
  120/121 [============================>.] - ETA: 0s - loss: 0.2723 - accuracy: 0.9057
  Epoch 11: val_accuracy did not improve from 0.76636
  121/121 [==============================] - 6s 52ms/step - loss: 0.2727 - accuracy: 0.9054 - val_loss: 0.9662 - val_accuracy: 0.7173
  Epoch 12/40
  120/121 [============================>.] - ETA: 0s - loss: 0.1859 - accuracy: 0.9315
  Epoch 12: val_accuracy improved from 0.76636 to 0.82944, saving model to best_model.h5
  121/121 [==============================] - 9s 71ms/step - loss: 0.1868 - accuracy: 0.9314 - val_loss: 0.5295 - val_accuracy: 0.8294
  Epoch 13/40
  120/121 [============================>.] - ETA: 0s - loss: 0.1668 - accuracy: 0.9398
  Epoch 13: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.1668 - accuracy: 0.9400 - val_loss: 0.8630 - val_accuracy: 0.7617
  Epoch 14/40
  120/121 [============================>.] - ETA: 0s - loss: 0.1245 - accuracy: 0.9604
  Epoch 14: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 51ms/step - loss: 0.1244 - accuracy: 0.9605 - val_loss: 7.8960 - val_accuracy: 0.2243
  Epoch 15/40
  120/121 [============================>.] - ETA: 0s - loss: 0.1046 - accuracy: 0.9672
  Epoch 15: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.1080 - accuracy: 0.9670 - val_loss: 0.7944 - val_accuracy: 0.7640
  Epoch 16/40
  120/121 [============================>.] - ETA: 0s - loss: 0.0991 - accuracy: 0.9680
  Epoch 16: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 51ms/step - loss: 0.0990 - accuracy: 0.9680 - val_loss: 0.7597 - val_accuracy: 0.7780
  Epoch 17/40
  120/121 [============================>.] - ETA: 0s - loss: 0.0788 - accuracy: 0.9742
  Epoch 17: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.0787 - accuracy: 0.9743 - val_loss: 0.6409 - val_accuracy: 0.8224
  Epoch 18/40
  121/121 [==============================] - ETA: 0s - loss: 0.0746 - accuracy: 0.9727
  Epoch 18: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.0746 - accuracy: 0.9727 - val_loss: 0.9586 - val_accuracy: 0.7383
  Epoch 19/40
  120/121 [============================>.] - ETA: 0s - loss: 0.0653 - accuracy: 0.9773
  Epoch 19: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 52ms/step - loss: 0.0652 - accuracy: 0.9774 - val_loss: 1.3687 - val_accuracy: 0.6706
  Epoch 20/40
  121/121 [==============================] - ETA: 0s - loss: 0.0584 - accuracy: 0.9784
  Epoch 20: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.0584 - accuracy: 0.9784 - val_loss: 1.8634 - val_accuracy: 0.6565
  Epoch 21/40
  120/121 [============================>.] - ETA: 0s - loss: 0.1058 - accuracy: 0.9643
  Epoch 21: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 50ms/step - loss: 0.1058 - accuracy: 0.9644 - val_loss: 1.1467 - val_accuracy: 0.7290
  Epoch 22/40
  120/121 [============================>.] - ETA: 0s - loss: 0.0926 - accuracy: 0.9688
  Epoch 22: val_accuracy did not improve from 0.82944
  121/121 [==============================] - 6s 53ms/step - loss: 0.0926 - accuracy: 0.9688 - val_loss: 4.7129 - val_accuracy: 0.4650
  Epoch 22: early stopping
  

Training stopped at epoch 24. The highest Validation accuracy was 0.8481 and validation loss 0.5729.

Model6 Accuracy vs Epoch curve

plt.plot(history6.history['accuracy'])
  plt.plot(history6.history['val_accuracy'])
  plt.title('model6 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

This model seems to be overfitting.

Evaluation of model6 model.evaluate using the Test data

# Evaluate the model on the test data
  score6 = model6.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score6[0])
  print('Test accuracy:', score6[1])
  

Test loss: 4.940274238586426
  Test accuracy: 0.42105263471603394
  

# Test Prediction
  y_pred = model6.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 13ms/step
  

# Test Accuracy
  accuracyScore6 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore6)
  

0.42105263157894735
  

Classification Report

# Compute the classification report
  cr6 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr6)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       1.00      0.46      0.63        50
                    Fat Hen       1.00      0.08      0.15        48
            Shepherds Purse       0.57      0.17      0.27        23
               Common wheat       0.00      0.00      0.00        22
           Common Chickweed       0.94      0.56      0.70        61
                   Charlock       1.00      0.10      0.19        39
                   Cleavers       1.00      0.07      0.13        29
          Scentless Mayweed       0.22      1.00      0.36        52
                 Sugar beet       1.00      0.03      0.05        38
                      Maize       0.33      0.64      0.43        22
                Black-grass       0.50      0.12      0.19        26
           Loose Silky-bent       0.54      0.91      0.67        65

                   accuracy                           0.42       475
                  macro avg       0.67      0.34      0.31       475
               weighted avg       0.72      0.42      0.37       475

  

The model seems to be overfitting as it has high accuracy on the training set but low accuracy on the test set. The precision, recall, and F1-score for each class also vary widely, with some classes having high scores and others having low scores. This suggests that the model is not performing well in some classes.

Scoresdf = Scoresdf.append({'model':'Model6','accuracy':accuracyScore6,'layers':'17','total parameters':'26,200,716', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Calculate Confusion matrix
  cm6 = confusion_matrix(normal_y_test, y_test_pred_classes)
  plt.figure(figsize=(8,6))
  sns.heatmap(cm5, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Overall, Model6 doesn't perform well for some classes and is overfitting.

Model 7

To continue trying to improve our model, we added more layers and modified the filter sizes.

• Conv2D layer with 32 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 32 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer to normalize the output of the previous layers.
• MaxPooling2D layer with a 2x2 pool size to reduce the spatial dimensions of the output.
• Dropout layer to randomly set a fraction of the input units to 0 to prevent overfitting.
• Conv2D layer with 64 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 64 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer.
• MaxPooling2D layer.
• Dropout layer.
• Conv2D layer with 128 filters, a 3x3 kernel size, and ReLU activation.
• Conv2D layer with 128 filters, a 3x3 kernel size, and ReLU activation.
• BatchNormalization layer.
• MaxPooling2D layer.
• Dropout layer.
• Flatten layer to convert the output of the previous layers to a 1D vector.
• Dense layer with 512 units and ReLU activation.
• BatchNormalization layer.
• Dropout layer.

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

#Model implementation
  model7 = Sequential()

  model7.add(Conv2D(32, (3, 3), activation='relu', input_shape=input_shape))
  model7.add(Conv2D(32, (3, 3), activation='relu'))
  model7.add(BatchNormalization())
  model7.add(MaxPooling2D(pool_size=(2, 2)))
  model7.add(Dropout(0.25))

  model7.add(Conv2D(64, (3, 3), activation='relu'))
  model7.add(Conv2D(64, (3, 3), activation='relu'))
  model7.add(BatchNormalization())
  model7.add(MaxPooling2D(pool_size=(2, 2)))
  model7.add(Dropout(0.25))

  model7.add(Conv2D(128, (3, 3), activation='relu'))
  model7.add(Conv2D(128, (3, 3), activation='relu'))
  model7.add(BatchNormalization())
  model7.add(MaxPooling2D(pool_size=(2, 2)))
  model7.add(Dropout(0.25))

  model7.add(Flatten())
  model7.add(Dense(512, activation='relu'))
  model7.add(BatchNormalization())
  model7.add(Dropout(0.5))
  model7.add(Dense(num_classes, activation='softmax'))

  model7.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])

  model7.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 32)      896

   conv2d_1 (Conv2D)           (None, 124, 124, 32)      9248

   batch_normalization (BatchN  (None, 124, 124, 32)     128
   ormalization)

   max_pooling2d (MaxPooling2D  (None, 62, 62, 32)       0
   )

   dropout (Dropout)           (None, 62, 62, 32)        0

   conv2d_2 (Conv2D)           (None, 60, 60, 64)        18496

   conv2d_3 (Conv2D)           (None, 58, 58, 64)        36928

   batch_normalization_1 (Batc  (None, 58, 58, 64)       256
   hNormalization)

   max_pooling2d_1 (MaxPooling  (None, 29, 29, 64)       0
   2D)

   dropout_1 (Dropout)         (None, 29, 29, 64)        0

   conv2d_4 (Conv2D)           (None, 27, 27, 128)       73856

   conv2d_5 (Conv2D)           (None, 25, 25, 128)       147584

   batch_normalization_2 (Batc  (None, 25, 25, 128)      512
   hNormalization)

   max_pooling2d_2 (MaxPooling  (None, 12, 12, 128)      0
   2D)

   dropout_2 (Dropout)         (None, 12, 12, 128)       0

   flatten (Flatten)           (None, 18432)             0

   dense (Dense)               (None, 512)               9437696

   batch_normalization_3 (Batc  (None, 512)              2048
   hNormalization)

   dropout_3 (Dropout)         (None, 512)               0

   dense_1 (Dense)             (None, 12)                6156

  =================================================================
  Total params: 9,733,804
  Trainable params: 9,732,332
  Non-trainable params: 1,472
  _________________________________________________________________
  

Model7 Architecture Visualization

#visualize model7 architecture
  plot_model(model7, to_file='model7_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize model7 architecture using visualkeras
  visualkeras.layered_view(model7, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 100 epochs and validation_split as 10%
  history7=model7.fit(X_train,
            y_train_onehot,
            epochs=100,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/100
  120/121 [============================>.] - ETA: 0s - loss: 1.9258 - accuracy: 0.4435
  Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 14s 73ms/step - loss: 1.9249 - accuracy: 0.4440 - val_loss: 23.2398 - val_accuracy: 0.1215
  Epoch 2/100
  120/121 [============================>.] - ETA: 0s - loss: 1.0223 - accuracy: 0.6716
  Epoch 2: val_accuracy did not improve from 0.12150
  121/121 [==============================] - 8s 63ms/step - loss: 1.0225 - accuracy: 0.6712 - val_loss: 18.6191 - val_accuracy: 0.1215
  Epoch 3/100
  120/121 [============================>.] - ETA: 0s - loss: 0.6844 - accuracy: 0.7745
  Epoch 3: val_accuracy did not improve from 0.12150
  121/121 [==============================] - 8s 63ms/step - loss: 0.6857 - accuracy: 0.7744 - val_loss: 21.4383 - val_accuracy: 0.1215
  Epoch 4/100
  120/121 [============================>.] - ETA: 0s - loss: 0.5225 - accuracy: 0.8263
  Epoch 4: val_accuracy improved from 0.12150 to 0.14252, saving model to best_model.h5
  121/121 [==============================] - 8s 70ms/step - loss: 0.5232 - accuracy: 0.8261 - val_loss: 7.6242 - val_accuracy: 0.1425
  Epoch 5/100
  120/121 [============================>.] - ETA: 0s - loss: 0.4009 - accuracy: 0.8602
  Epoch 5: val_accuracy improved from 0.14252 to 0.18925, saving model to best_model.h5
  121/121 [==============================] - 8s 68ms/step - loss: 0.4030 - accuracy: 0.8594 - val_loss: 5.4088 - val_accuracy: 0.1893
  Epoch 6/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3923 - accuracy: 0.8622
  Epoch 6: val_accuracy improved from 0.18925 to 0.67290, saving model to best_model.h5
  121/121 [==============================] - 14s 113ms/step - loss: 0.3940 - accuracy: 0.8615 - val_loss: 1.0581 - val_accuracy: 0.6729
  Epoch 7/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3071 - accuracy: 0.8935
  Epoch 7: val_accuracy improved from 0.67290 to 0.79673, saving model to best_model.h5
  121/121 [==============================] - 8s 69ms/step - loss: 0.3068 - accuracy: 0.8937 - val_loss: 0.7298 - val_accuracy: 0.7967
  Epoch 8/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2381 - accuracy: 0.9133
  Epoch 8: val_accuracy did not improve from 0.79673
  121/121 [==============================] - 8s 62ms/step - loss: 0.2386 - accuracy: 0.9132 - val_loss: 1.3537 - val_accuracy: 0.6612
  Epoch 9/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1960 - accuracy: 0.9320
  Epoch 9: val_accuracy improved from 0.79673 to 0.79907, saving model to best_model.h5
  121/121 [==============================] - 8s 68ms/step - loss: 0.1963 - accuracy: 0.9319 - val_loss: 0.7284 - val_accuracy: 0.7991
  Epoch 10/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1645 - accuracy: 0.9409
  Epoch 10: val_accuracy did not improve from 0.79907
  121/121 [==============================] - 8s 62ms/step - loss: 0.1655 - accuracy: 0.9402 - val_loss: 0.8174 - val_accuracy: 0.7617
  Epoch 11/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1333 - accuracy: 0.9555
  Epoch 11: val_accuracy improved from 0.79907 to 0.85047, saving model to best_model.h5
  121/121 [==============================] - 8s 67ms/step - loss: 0.1351 - accuracy: 0.9550 - val_loss: 0.4578 - val_accuracy: 0.8505
  Epoch 12/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1134 - accuracy: 0.9591
  Epoch 12: val_accuracy did not improve from 0.85047
  121/121 [==============================] - 8s 63ms/step - loss: 0.1148 - accuracy: 0.9584 - val_loss: 1.0042 - val_accuracy: 0.7593
  Epoch 13/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1403 - accuracy: 0.9497
  Epoch 13: val_accuracy did not improve from 0.85047
  121/121 [==============================] - 8s 65ms/step - loss: 0.1414 - accuracy: 0.9493 - val_loss: 0.6063 - val_accuracy: 0.8458
  Epoch 14/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1181 - accuracy: 0.9583
  Epoch 14: val_accuracy did not improve from 0.85047
  121/121 [==============================] - 8s 64ms/step - loss: 0.1183 - accuracy: 0.9581 - val_loss: 1.3537 - val_accuracy: 0.6682
  Epoch 15/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1149 - accuracy: 0.9628
  Epoch 15: val_accuracy did not improve from 0.85047
  121/121 [==============================] - 8s 63ms/step - loss: 0.1179 - accuracy: 0.9623 - val_loss: 1.4002 - val_accuracy: 0.6799
  Epoch 16/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0929 - accuracy: 0.9680
  Epoch 16: val_accuracy did not improve from 0.85047
  121/121 [==============================] - 8s 64ms/step - loss: 0.0929 - accuracy: 0.9680 - val_loss: 0.7927 - val_accuracy: 0.8201
  Epoch 17/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0706 - accuracy: 0.9747
  Epoch 17: val_accuracy improved from 0.85047 to 0.85280, saving model to best_model.h5
  121/121 [==============================] - 9s 72ms/step - loss: 0.0705 - accuracy: 0.9748 - val_loss: 0.5961 - val_accuracy: 0.8528
  Epoch 18/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0556 - accuracy: 0.9815
  Epoch 18: val_accuracy did not improve from 0.85280
  121/121 [==============================] - 8s 64ms/step - loss: 0.0561 - accuracy: 0.9813 - val_loss: 1.0609 - val_accuracy: 0.7313
  Epoch 19/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1063 - accuracy: 0.9648
  Epoch 19: val_accuracy did not improve from 0.85280
  121/121 [==============================] - 8s 64ms/step - loss: 0.1063 - accuracy: 0.9649 - val_loss: 0.5733 - val_accuracy: 0.8458
  Epoch 20/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0721 - accuracy: 0.9755
  Epoch 20: val_accuracy did not improve from 0.85280
  121/121 [==============================] - 8s 63ms/step - loss: 0.0728 - accuracy: 0.9753 - val_loss: 0.5768 - val_accuracy: 0.8528
  Epoch 21/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0902 - accuracy: 0.9714
  Epoch 21: val_accuracy did not improve from 0.85280
  121/121 [==============================] - 8s 64ms/step - loss: 0.0900 - accuracy: 0.9714 - val_loss: 0.6353 - val_accuracy: 0.8248
  Epoch 21: early stopping
  

Training stopped at epoch 27. The highest validation accuracy was 0.8578 with validation loss of 0.25

Model7 Accuracy vs Epoch curve

plt.plot(history7.history['accuracy'])
  plt.plot(history7.history['val_accuracy'])
  plt.title('model7 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

Model is overfitting

Model7 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
  score7 = model7.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score7[0])
  print('Test accuracy:', score7[1])
  

Test loss: 0.7704196572303772
  Test accuracy: 0.7957894802093506
  

# Test Prediction
  y_pred = model7.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 13ms/step
  

# Test Accuracy
  accuracyScore7 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore7)
  

0.7957894736842105
  

Classification Report

# Compute the classification report
  cr7 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr7)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.89      0.80      0.84        50
                    Fat Hen       0.75      0.96      0.84        48
            Shepherds Purse       0.78      0.61      0.68        23
               Common wheat       0.76      0.59      0.67        22
           Common Chickweed       0.94      0.80      0.87        61
                   Charlock       0.84      0.95      0.89        39
                   Cleavers       0.87      0.90      0.88        29
          Scentless Mayweed       0.85      0.87      0.86        52
                 Sugar beet       0.82      0.84      0.83        38
                      Maize       0.83      0.68      0.75        22
                Black-grass       0.38      0.46      0.41        26
           Loose Silky-bent       0.74      0.75      0.75        65

                   accuracy                           0.80       475
                  macro avg       0.79      0.77      0.77       475
               weighted avg       0.81      0.80      0.80       475

  

The accuracy of the model is very low at 0.24. For most of the classes, precision, recall, and f1-score are also very low. This suggests that the model is not able to classify the images correctly, and the predictions made by the model are almost random. Therefore, it is not a valid model.

Scoresdf = Scoresdf.append({'model':'Model7','accuracy':accuracyScore7,'layers':'20','total parameters':'9,733,804', 'neurons':'512', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Compute the Confusion matrix
  cm7 = confusion_matrix(normal_y_test, y_test_pred_classes)

  plt.figure(figsize=(8,6))
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  sns.heatmap(cm7, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.show()
  

Model7 is not a valid model.

Model8

Continuing trying to improve the previous model, we changed the number of layers and neurons.

• Convolutional layer, 64 filters, kernel (3,3), ReLU
• Batch normalization
• MaxPooling2D
• Convolutional layer, 128 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 256 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 128 filters, kernel (3,3).
• Batch normalization
• MaxPooling2D
• Convolutional layer, 64 filters, kernel (5,5).
• Batch normalization
• MaxPooling2D
• Flatten Layer
• Dense Layer, 256 neurons
• BatchNormalization
• Drop Layer 0.5

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

Model Implementation

# initialized a sequential model
  model8 = Sequential()

  model8.add(Conv2D(64, kernel_size=(3,3), activation='relu', input_shape=input_shape))
  model8.add(BatchNormalization())
  model8.add(MaxPooling2D(pool_size=(2,2)))

  model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model8.add(BatchNormalization())
  model8.add(MaxPooling2D(pool_size=(2,2)))

  model8.add(Conv2D(256, kernel_size=(3,3), activation='relu'))
  model8.add(BatchNormalization())
  model8.add(MaxPooling2D(pool_size=(2,2)))

  model8.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model8.add(BatchNormalization())
  model8.add(MaxPooling2D(pool_size=(2,2)))

  model8.add(Conv2D(64, kernel_size=(5,5), activation='relu'))
  model8.add(BatchNormalization())
  model8.add(MaxPooling2D(pool_size=(2,2)))

  model8.add(Flatten())
  model8.add(Dense(256, activation='relu'))
  model8.add(BatchNormalization())
  model8.add(Dropout(0.5))
  model8.add(Dense(num_classes, activation='softmax'))

  # Compile the model
  model8.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

  # Print the model summary
  model8.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 126, 126, 64)      1792

   batch_normalization (BatchN  (None, 126, 126, 64)     256
   ormalization)

   max_pooling2d (MaxPooling2D  (None, 63, 63, 64)       0
   )

   conv2d_1 (Conv2D)           (None, 61, 61, 128)       73856

   batch_normalization_1 (Batc  (None, 61, 61, 128)      512
   hNormalization)

   max_pooling2d_1 (MaxPooling  (None, 30, 30, 128)      0
   2D)

   conv2d_2 (Conv2D)           (None, 28, 28, 256)       295168

   batch_normalization_2 (Batc  (None, 28, 28, 256)      1024
   hNormalization)

   max_pooling2d_2 (MaxPooling  (None, 14, 14, 256)      0
   2D)

   conv2d_3 (Conv2D)           (None, 12, 12, 128)       295040

   batch_normalization_3 (Batc  (None, 12, 12, 128)      512
   hNormalization)

   max_pooling2d_3 (MaxPooling  (None, 6, 6, 128)        0
   2D)

   conv2d_4 (Conv2D)           (None, 2, 2, 64)          204864

   batch_normalization_4 (Batc  (None, 2, 2, 64)         256
   hNormalization)

   max_pooling2d_4 (MaxPooling  (None, 1, 1, 64)         0
   2D)

   flatten (Flatten)           (None, 64)                0

   dense (Dense)               (None, 256)               16640

   batch_normalization_5 (Batc  (None, 256)              1024
   hNormalization)

   dropout (Dropout)           (None, 256)               0

   dense_1 (Dense)             (None, 12)                3084

  =================================================================
  Total params: 894,028
  Trainable params: 892,236
  Non-trainable params: 1,792
  _________________________________________________________________
  

Model8 Architecture Visualization

#visualize model4 architecture
  plot_model(model8, to_file='model8_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize model8 architecture using visualkeras
  visualkeras.layered_view(model8, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 100 epochs and validation_split as 10%
  history8=model8.fit(X_train,
            y_train_onehot,
            epochs=100,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/100
  121/121 [==============================] - ETA: 0s - loss: 1.9141 - accuracy: 0.4328
  Epoch 1: val_accuracy improved from -inf to 0.11916, saving model to best_model.h5
  121/121 [==============================] - 12s 51ms/step - loss: 1.9141 - accuracy: 0.4328 - val_loss: 6.6545 - val_accuracy: 0.1192
  Epoch 2/100
  121/121 [==============================] - ETA: 0s - loss: 1.0103 - accuracy: 0.6740
  Epoch 2: val_accuracy improved from 0.11916 to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 6s 49ms/step - loss: 1.0103 - accuracy: 0.6740 - val_loss: 13.4881 - val_accuracy: 0.1215
  Epoch 3/100
  120/121 [============================>.] - ETA: 0s - loss: 0.7239 - accuracy: 0.7560
  Epoch 3: val_accuracy improved from 0.12150 to 0.12850, saving model to best_model.h5
  121/121 [==============================] - 6s 49ms/step - loss: 0.7257 - accuracy: 0.7554 - val_loss: 9.6117 - val_accuracy: 0.1285
  Epoch 4/100
  120/121 [============================>.] - ETA: 0s - loss: 0.5796 - accuracy: 0.8023
  Epoch 4: val_accuracy did not improve from 0.12850
  121/121 [==============================] - 6s 48ms/step - loss: 0.5814 - accuracy: 0.8019 - val_loss: 8.5606 - val_accuracy: 0.1285
  Epoch 5/100
  120/121 [============================>.] - ETA: 0s - loss: 0.4688 - accuracy: 0.8372
  Epoch 5: val_accuracy improved from 0.12850 to 0.43692, saving model to best_model.h5
  121/121 [==============================] - 6s 49ms/step - loss: 0.4694 - accuracy: 0.8368 - val_loss: 2.2889 - val_accuracy: 0.4369
  Epoch 6/100
  121/121 [==============================] - ETA: 0s - loss: 0.4436 - accuracy: 0.8417
  Epoch 6: val_accuracy improved from 0.43692 to 0.51636, saving model to best_model.h5
  121/121 [==============================] - 6s 49ms/step - loss: 0.4436 - accuracy: 0.8417 - val_loss: 1.5959 - val_accuracy: 0.5164
  Epoch 7/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8878
  Epoch 7: val_accuracy improved from 0.51636 to 0.69860, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 0.3200 - accuracy: 0.8880 - val_loss: 1.0497 - val_accuracy: 0.6986
  Epoch 8/100
  121/121 [==============================] - ETA: 0s - loss: 0.3005 - accuracy: 0.8942
  Epoch 8: val_accuracy did not improve from 0.69860
  121/121 [==============================] - 6s 48ms/step - loss: 0.3005 - accuracy: 0.8942 - val_loss: 5.7750 - val_accuracy: 0.2056
  Epoch 9/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2668 - accuracy: 0.9023
  Epoch 9: val_accuracy did not improve from 0.69860
  121/121 [==============================] - 6s 49ms/step - loss: 0.2665 - accuracy: 0.9025 - val_loss: 7.3724 - val_accuracy: 0.2126
  Epoch 10/100
  121/121 [==============================] - ETA: 0s - loss: 0.2086 - accuracy: 0.9194
  Epoch 10: val_accuracy improved from 0.69860 to 0.73598, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 0.2086 - accuracy: 0.9194 - val_loss: 0.8748 - val_accuracy: 0.7360
  Epoch 11/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2114 - accuracy: 0.9227
  Epoch 11: val_accuracy improved from 0.73598 to 0.82944, saving model to best_model.h5
  121/121 [==============================] - 6s 48ms/step - loss: 0.2125 - accuracy: 0.9223 - val_loss: 0.5081 - val_accuracy: 0.8294
  Epoch 12/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2021 - accuracy: 0.9268
  Epoch 12: val_accuracy improved from 0.82944 to 0.84813, saving model to best_model.h5
  121/121 [==============================] - 6s 52ms/step - loss: 0.2023 - accuracy: 0.9267 - val_loss: 0.4966 - val_accuracy: 0.8481
  Epoch 13/100
  121/121 [==============================] - ETA: 0s - loss: 0.1775 - accuracy: 0.9363
  Epoch 13: val_accuracy did not improve from 0.84813
  121/121 [==============================] - 6s 49ms/step - loss: 0.1775 - accuracy: 0.9363 - val_loss: 1.5754 - val_accuracy: 0.6519
  Epoch 14/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1478 - accuracy: 0.9443
  Epoch 14: val_accuracy did not improve from 0.84813
  121/121 [==============================] - 6s 49ms/step - loss: 0.1481 - accuracy: 0.9441 - val_loss: 0.5397 - val_accuracy: 0.8435
  Epoch 15/100
  121/121 [==============================] - ETA: 0s - loss: 0.1143 - accuracy: 0.9615
  Epoch 15: val_accuracy did not improve from 0.84813
  121/121 [==============================] - 6s 50ms/step - loss: 0.1143 - accuracy: 0.9615 - val_loss: 1.3775 - val_accuracy: 0.6916
  Epoch 16/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2131 - accuracy: 0.9289
  Epoch 16: val_accuracy did not improve from 0.84813
  121/121 [==============================] - 6s 47ms/step - loss: 0.2155 - accuracy: 0.9288 - val_loss: 1.4203 - val_accuracy: 0.6799
  Epoch 17/100
  121/121 [==============================] - ETA: 0s - loss: 0.1556 - accuracy: 0.9480
  Epoch 17: val_accuracy improved from 0.84813 to 0.86682, saving model to best_model.h5
  121/121 [==============================] - 6s 52ms/step - loss: 0.1556 - accuracy: 0.9480 - val_loss: 0.4611 - val_accuracy: 0.8668
  Epoch 18/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0900 - accuracy: 0.9701
  Epoch 18: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 48ms/step - loss: 0.0899 - accuracy: 0.9701 - val_loss: 0.8230 - val_accuracy: 0.7477
  Epoch 19/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0837 - accuracy: 0.9714
  Epoch 19: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 49ms/step - loss: 0.0856 - accuracy: 0.9711 - val_loss: 1.8179 - val_accuracy: 0.6379
  Epoch 20/100
  121/121 [==============================] - ETA: 0s - loss: 0.1230 - accuracy: 0.9561
  Epoch 20: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 50ms/step - loss: 0.1230 - accuracy: 0.9561 - val_loss: 0.7045 - val_accuracy: 0.8061
  Epoch 21/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1514 - accuracy: 0.9479
  Epoch 21: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 48ms/step - loss: 0.1512 - accuracy: 0.9480 - val_loss: 0.6942 - val_accuracy: 0.7944
  Epoch 22/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0827 - accuracy: 0.9693
  Epoch 22: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 50ms/step - loss: 0.0827 - accuracy: 0.9693 - val_loss: 0.6314 - val_accuracy: 0.8341
  Epoch 23/100
  121/121 [==============================] - ETA: 0s - loss: 0.0603 - accuracy: 0.9800
  Epoch 23: val_accuracy did not improve from 0.86682
  121/121 [==============================] - 6s 49ms/step - loss: 0.0603 - accuracy: 0.9800 - val_loss: 2.7126 - val_accuracy: 0.5047
  Epoch 24/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0530 - accuracy: 0.9846
  Epoch 24: val_accuracy improved from 0.86682 to 0.89720, saving model to best_model.h5
  121/121 [==============================] - 6s 49ms/step - loss: 0.0538 - accuracy: 0.9844 - val_loss: 0.3790 - val_accuracy: 0.8972
  Epoch 25/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1026 - accuracy: 0.9643
  Epoch 25: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 49ms/step - loss: 0.1034 - accuracy: 0.9641 - val_loss: 1.1961 - val_accuracy: 0.6776
  Epoch 26/100
  121/121 [==============================] - ETA: 0s - loss: 0.1000 - accuracy: 0.9670
  Epoch 26: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 48ms/step - loss: 0.1000 - accuracy: 0.9670 - val_loss: 0.5659 - val_accuracy: 0.8621
  Epoch 27/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0665 - accuracy: 0.9784
  Epoch 27: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 49ms/step - loss: 0.0669 - accuracy: 0.9782 - val_loss: 0.5263 - val_accuracy: 0.8668
  Epoch 28/100
  121/121 [==============================] - ETA: 0s - loss: 0.0534 - accuracy: 0.9839
  Epoch 28: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 48ms/step - loss: 0.0534 - accuracy: 0.9839 - val_loss: 0.5661 - val_accuracy: 0.8668
  Epoch 29/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0806 - accuracy: 0.9729
  Epoch 29: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 47ms/step - loss: 0.0826 - accuracy: 0.9724 - val_loss: 1.1458 - val_accuracy: 0.7033
  Epoch 30/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0935 - accuracy: 0.9682
  Epoch 30: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 49ms/step - loss: 0.0937 - accuracy: 0.9680 - val_loss: 0.6199 - val_accuracy: 0.8528
  Epoch 31/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0455 - accuracy: 0.9870
  Epoch 31: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 47ms/step - loss: 0.0459 - accuracy: 0.9867 - val_loss: 0.6134 - val_accuracy: 0.8458
  Epoch 32/100
  121/121 [==============================] - ETA: 0s - loss: 0.0428 - accuracy: 0.9857
  Epoch 32: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 48ms/step - loss: 0.0428 - accuracy: 0.9857 - val_loss: 0.7207 - val_accuracy: 0.8575
  Epoch 33/100
  121/121 [==============================] - ETA: 0s - loss: 0.0531 - accuracy: 0.9802
  Epoch 33: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 49ms/step - loss: 0.0531 - accuracy: 0.9802 - val_loss: 0.7820 - val_accuracy: 0.7757
  Epoch 34/100
  120/121 [============================>.] - ETA: 0s - loss: 0.0499 - accuracy: 0.9828
  Epoch 34: val_accuracy did not improve from 0.89720
  121/121 [==============================] - 6s 48ms/step - loss: 0.0509 - accuracy: 0.9826 - val_loss: 1.0555 - val_accuracy: 0.7687
  Epoch 34: early stopping
  

Training stopped at epoch 20. The highest validation accuracy was 0.8916 with a validation loss of 0.3837.

Model8 Accuracy vs Epoch curve

plt.plot(history8.history['accuracy'])
  plt.plot(history8.history['val_accuracy'])
  plt.title('model8 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

The model overfits.

Model8 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
  score8 = model8.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score8[0])
  print('Test accuracy:', score8[1])
  

Test loss: 0.9670275449752808
  Test accuracy: 0.7873684167861938
  

# Test Prediction
  y_pred = model8.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 13ms/step
  

# Test Accuracy
  accuracyScore8 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore8)
  

0.7873684210526316
  

Classification Report

# Compute the classification report
  cr8 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr8)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.90      0.92      0.91        50
                    Fat Hen       0.94      0.96      0.95        48
            Shepherds Purse       0.89      0.35      0.50        23
               Common wheat       0.80      0.73      0.76        22
           Common Chickweed       0.63      0.98      0.77        61
                   Charlock       0.95      0.90      0.92        39
                   Cleavers       0.81      0.90      0.85        29
          Scentless Mayweed       0.97      0.63      0.77        52
                 Sugar beet       0.80      0.84      0.82        38
                      Maize       1.00      0.59      0.74        22
                Black-grass       0.39      0.62      0.48        26
           Loose Silky-bent       0.80      0.66      0.72        65

                   accuracy                           0.79       475
                  macro avg       0.82      0.76      0.77       475
               weighted avg       0.83      0.79      0.79       475

  

Overall the accuracy is low at 0.43. We have developed better models earlier.

Scoresdf = Scoresdf.append({'model':'Model8','accuracy':accuracyScore8,'layers':'20','total parameters':'894,028', 'neurons':'256', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Compute Confusion matrix
  cm8 = confusion_matrix(normal_y_test, y_test_pred_classes)
  plt.figure(figsize=(8,6))
  sns.heatmap(cm8, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

From the confusion matrix we can observe that Model8 is unable to predict many classes correctly at all. Overall, this is not a good model.

Model 9

backend.clear_session()
  #Fixing the seed for random number generators so that we can ensure we receive the same output everytime
  np.random.seed(42)
  import random
  random.seed(42)
  tf.random.set_seed(42)
  

This will be our final model trying to improve our previous models. Again, we will modify the number of layers, filter size and neurons.

Model Implementation

# Intializing a sequential model
  model9 = Sequential()

  # Adding first conv layer with 64 filters and kernel size 3x3 , padding 'same' provides the output size same as the input size
  # Input_shape denotes input image dimension of MNIST images
  model9.add(Conv2D(64, (3, 3), activation='relu', padding="same", input_shape=input_shape))
  model9.add(Conv2D(64, (3, 3), activation='relu', padding="same"))
  # Adding max pooling to reduce the size of output of first conv layer
  model9.add(MaxPooling2D((2, 2), padding = 'same'))

  model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
  model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
  model9.add(MaxPooling2D((2, 2), padding = 'same'))
  model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
  model9.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
  model9.add(MaxPooling2D((2, 2), padding = 'same'))

  # flattening the output of the conv layer after max pooling to make it ready for creating dense connections
  model9.add(Flatten())

  # Adding a fully connected dense layer with 100 neurons
  model9.add(Dense(128, activation='relu'))
  model9.add(Dropout(0.25))
  model9.add(Dense(64, activation='relu'))
  # Adding the output layer with 10 neurons and activation functions as softmax since this is a multi-class classification problem
  model9.add(Dense(num_classes, activation='softmax'))

  # Using Adam Optimizer
  opt = Adam()

  # Compile model
  model9.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  

model9.summary()
  

Model: "sequential"
  _________________________________________________________________
   Layer (type)                Output Shape              Param #
  =================================================================
   conv2d (Conv2D)             (None, 128, 128, 64)      1792

   conv2d_1 (Conv2D)           (None, 128, 128, 64)      36928

   max_pooling2d (MaxPooling2D  (None, 64, 64, 64)       0
   )

   conv2d_2 (Conv2D)           (None, 64, 64, 32)        18464

   conv2d_3 (Conv2D)           (None, 64, 64, 32)        9248

   max_pooling2d_1 (MaxPooling  (None, 32, 32, 32)       0
   2D)

   conv2d_4 (Conv2D)           (None, 32, 32, 32)        9248

   conv2d_5 (Conv2D)           (None, 32, 32, 32)        9248

   max_pooling2d_2 (MaxPooling  (None, 16, 16, 32)       0
   2D)

   flatten (Flatten)           (None, 8192)              0

   dense (Dense)               (None, 128)               1048704

   dropout (Dropout)           (None, 128)               0

   dense_1 (Dense)             (None, 64)                8256

   dense_2 (Dense)             (None, 12)                780

  =================================================================
  Total params: 1,142,668
  Trainable params: 1,142,668
  Non-trainable params: 0
  _________________________________________________________________
  

Model9 Architecture Visualization

#visualize model9 architecture
  plot_model(model9, to_file='model9_plot.png', show_shapes=True, show_layer_names=True)
  

#visualize model9 architecture using visualkeras
  visualkeras.layered_view(model9, legend=True)
  

Early Stopping

Use early stopping to terminate the epochs if the validation loss continues to increase or flats. We are monitoring for early stopping the validation loss (so mode='min'). Also, we set patience to 10. And we save the epoch with the highest validation accuracy.

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=10)
  mc = ModelCheckpoint('best_model.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
  # Fitting the model with 100 epochs and validation_split as 10%
  history9=model9.fit(X_train,
            y_train_onehot,
            epochs=100,
            batch_size=32,validation_split=0.10,callbacks=[es, mc])
  

Epoch 1/100
  120/121 [============================>.] - ETA: 0s - loss: 2.4315 - accuracy: 0.1240
  Epoch 1: val_accuracy improved from -inf to 0.12150, saving model to best_model.h5
  121/121 [==============================] - 10s 55ms/step - loss: 2.4312 - accuracy: 0.1237 - val_loss: 2.4521 - val_accuracy: 0.1215
  Epoch 2/100
  120/121 [============================>.] - ETA: 0s - loss: 2.0708 - accuracy: 0.2891
  Epoch 2: val_accuracy improved from 0.12150 to 0.35748, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 2.0710 - accuracy: 0.2893 - val_loss: 1.7058 - val_accuracy: 0.3575
  Epoch 3/100
  120/121 [============================>.] - ETA: 0s - loss: 1.5718 - accuracy: 0.4453
  Epoch 3: val_accuracy improved from 0.35748 to 0.48131, saving model to best_model.h5
  121/121 [==============================] - 6s 52ms/step - loss: 1.5718 - accuracy: 0.4453 - val_loss: 1.5090 - val_accuracy: 0.4813
  Epoch 4/100
  120/121 [============================>.] - ETA: 0s - loss: 1.3208 - accuracy: 0.5378
  Epoch 4: val_accuracy improved from 0.48131 to 0.59112, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 1.3205 - accuracy: 0.5378 - val_loss: 1.1847 - val_accuracy: 0.5911
  Epoch 5/100
  121/121 [==============================] - ETA: 0s - loss: 1.0526 - accuracy: 0.6356
  Epoch 5: val_accuracy improved from 0.59112 to 0.62617, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 1.0526 - accuracy: 0.6356 - val_loss: 1.0737 - val_accuracy: 0.6262
  Epoch 6/100
  121/121 [==============================] - ETA: 0s - loss: 0.8805 - accuracy: 0.6868
  Epoch 6: val_accuracy improved from 0.62617 to 0.67290, saving model to best_model.h5
  121/121 [==============================] - 6s 51ms/step - loss: 0.8805 - accuracy: 0.6868 - val_loss: 0.9630 - val_accuracy: 0.6729
  Epoch 7/100
  120/121 [============================>.] - ETA: 0s - loss: 0.7282 - accuracy: 0.7346
  Epoch 7: val_accuracy improved from 0.67290 to 0.68224, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 0.7284 - accuracy: 0.7349 - val_loss: 0.9714 - val_accuracy: 0.6822
  Epoch 8/100
  120/121 [============================>.] - ETA: 0s - loss: 0.5953 - accuracy: 0.7810
  Epoch 8: val_accuracy improved from 0.68224 to 0.71262, saving model to best_model.h5
  121/121 [==============================] - 6s 53ms/step - loss: 0.5949 - accuracy: 0.7809 - val_loss: 0.8997 - val_accuracy: 0.7126
  Epoch 9/100
  121/121 [==============================] - ETA: 0s - loss: 0.4768 - accuracy: 0.8095
  Epoch 9: val_accuracy improved from 0.71262 to 0.71495, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 0.4768 - accuracy: 0.8095 - val_loss: 1.0595 - val_accuracy: 0.7150
  Epoch 10/100
  121/121 [==============================] - ETA: 0s - loss: 0.4320 - accuracy: 0.8303
  Epoch 10: val_accuracy improved from 0.71495 to 0.73364, saving model to best_model.h5
  121/121 [==============================] - 6s 50ms/step - loss: 0.4320 - accuracy: 0.8303 - val_loss: 1.0687 - val_accuracy: 0.7336
  Epoch 11/100
  120/121 [============================>.] - ETA: 0s - loss: 0.4239 - accuracy: 0.8435
  Epoch 11: val_accuracy did not improve from 0.73364
  121/121 [==============================] - 6s 50ms/step - loss: 0.4241 - accuracy: 0.8433 - val_loss: 1.0077 - val_accuracy: 0.7313
  Epoch 12/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3202 - accuracy: 0.8776
  Epoch 12: val_accuracy did not improve from 0.73364
  121/121 [==============================] - 6s 48ms/step - loss: 0.3203 - accuracy: 0.8773 - val_loss: 1.1118 - val_accuracy: 0.7220
  Epoch 13/100
  120/121 [============================>.] - ETA: 0s - loss: 0.3107 - accuracy: 0.8802
  Epoch 13: val_accuracy improved from 0.73364 to 0.75467, saving model to best_model.h5
  121/121 [==============================] - 6s 53ms/step - loss: 0.3113 - accuracy: 0.8799 - val_loss: 0.9295 - val_accuracy: 0.7547
  Epoch 14/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2699 - accuracy: 0.9031
  Epoch 14: val_accuracy did not improve from 0.75467
  121/121 [==============================] - 6s 49ms/step - loss: 0.2700 - accuracy: 0.9030 - val_loss: 1.0213 - val_accuracy: 0.7383
  Epoch 15/100
  120/121 [============================>.] - ETA: 0s - loss: 0.2276 - accuracy: 0.9146
  Epoch 15: val_accuracy did not improve from 0.75467
  121/121 [==============================] - 6s 51ms/step - loss: 0.2274 - accuracy: 0.9147 - val_loss: 1.1179 - val_accuracy: 0.7336
  Epoch 16/100
  121/121 [==============================] - ETA: 0s - loss: 0.2142 - accuracy: 0.9254
  Epoch 16: val_accuracy did not improve from 0.75467
  121/121 [==============================] - 6s 50ms/step - loss: 0.2142 - accuracy: 0.9254 - val_loss: 1.1452 - val_accuracy: 0.7173
  Epoch 17/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1628 - accuracy: 0.9440
  Epoch 17: val_accuracy did not improve from 0.75467
  121/121 [==============================] - 6s 50ms/step - loss: 0.1625 - accuracy: 0.9441 - val_loss: 1.0877 - val_accuracy: 0.7360
  Epoch 18/100
  120/121 [============================>.] - ETA: 0s - loss: 0.1426 - accuracy: 0.9451
  Epoch 18: val_accuracy did not improve from 0.75467
  121/121 [==============================] - 6s 52ms/step - loss: 0.1425 - accuracy: 0.9452 - val_loss: 1.2663 - val_accuracy: 0.7430
  Epoch 18: early stopping
  

Training stopped at epoch 20. The highest validation accuracy was 0.79907 with a validation loss of 1.145.

Model9 Accuracy vs Epoch curve

plt.plot(history9.history['accuracy'])
  plt.plot(history9.history['val_accuracy'])
  plt.title('model9 accuracy')
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'validation'], loc='upper left')
  plt.show()
  

The model overfits.

Model9 Evaluation of model model.evaluate using the Test data

# Evaluate the model on the test data
  score9 = model9.evaluate(X_test, y_test_onehot, verbose=0)
  print('Test loss:', score8[0])
  print('Test accuracy:', score8[1])
  

Test loss: 0.9670275449752808
  Test accuracy: 0.7873684167861938
  

# Test Prediction
  y_pred = model9.predict(X_test)
  y_test_pred_classes = np.argmax(y_pred, axis=1)
  normal_y_test = np.argmax(y_test_onehot, axis=1)
  

15/15 [==============================] - 0s 14ms/step
  

# Test Accuracy
  accuracyScore9 = accuracy_score((normal_y_test), y_test_pred_classes)
  print(accuracyScore8)
  

0.7873684210526316
  

Classification Report

# Compute the classification report
  cr9 = classification_report(normal_y_test, y_test_pred_classes, target_names=name_classes)

  print('Classification report:\n',cr9)
  

Classification report:
                              precision    recall  f1-score   support

  Small-flowered Cranesbill       0.90      0.86      0.88        50
                    Fat Hen       0.73      0.85      0.79        48
            Shepherds Purse       0.53      0.35      0.42        23
               Common wheat       0.64      0.64      0.64        22
           Common Chickweed       0.75      0.92      0.82        61
                   Charlock       0.77      0.87      0.82        39
                   Cleavers       0.81      0.86      0.83        29
          Scentless Mayweed       0.71      0.65      0.68        52
                 Sugar beet       0.77      0.63      0.70        38
                      Maize       0.65      0.50      0.56        22
                Black-grass       0.36      0.35      0.35        26
           Loose Silky-bent       0.70      0.68      0.69        65

                   accuracy                           0.72       475
                  macro avg       0.69      0.68      0.68       475
               weighted avg       0.72      0.72      0.72       475

  

Model9 with four convolutional layers indicates that the overall performance is good, with an accuracy of 78%.

Scoresdf = Scoresdf.append({'model':'Model9','accuracy':accuracyScore9,'layers':'14','total parameters':'1,142,668', 'neurons':'64', 'optimizer':'Adam'},ignore_index=True)
  

Confusion Matrix

# Compute Confusion matrix
  cm9 = confusion_matrix(normal_y_test, y_test_pred_classes)
  plt.figure(figsize=(8,6))
  sns.heatmap(cm9, xticklabels=name_classes, yticklabels=name_classes, annot=True)
  plt.xlabel("False Negatives (FN)")
  plt.ylabel("False Positives (FP)")
  plt.show()
  

Model9 performs well but not as good as Model4.

Model Improvement and Model Comparison

We tried to improve our model by implementing 9 models. One approach to implement a model is by changing the hyperparameters like optimizer, number of neurons, number of filters, adding convolution layers, batchnormalizers, changing the dropout rate, etc.

We performed these to improve our model's performance:

• Architecture Modifications: Modifying the architecture of the CNN by adding or removing layers, changing the number of filters or neurons in each layer, or using different activation functions can improve performance. We implemented diverse models this way.

• Hyperparameter Tuning: Optimizing the hyperparameters of the model, such as learning rate, batch size, and optimizer, can improve performance. This can be done using techniques such as grid search or random search. We changed the optimizer (Adam, SGD).

• Regularization: Regularization techniques such as dropout and L2 regularization can prevent overfitting and improve generalization performance. We performed dropout.

• Batch Normalization: Batch normalization is a technique that normalizes the inputs of each layer to prevent internal covariate shift. This can improve performance and reduce the number of epochs needed for training. We used batch Normalization.

• Early Stopping: Early stopping is a technique that stops training the model when the validation loss stops improving. This prevents overfitting and improves generalization performance. We used Early stopping.

We didn't implement:

• Data Augmentation: Increasing the amount of data available for training the model can improve performance. Data augmentation techniques such as flipping, rotating, scaling, and cropping images can create new variations of the same images and help the model to generalize better. However, this may require computational power so it may be expensive to perform.

• Transfer Learning: Pretrained models can be used as a starting point for training a new model. Transfer learning involves using the weights of a model trained on a large dataset, such as ImageNet, to initialize the weights of a new model. This can speed up training and improve performance.

Ensembling: Ensemble methods involve combining the predictions of multiple models to improve performance. This can be done by training multiple CNN models with different hyperparameters or architectures and combining their predictions.

From Model Improvement we came up with 9 different models which are compared below:

Scoresdf
  

	model	accuracy	layers	total parameters	neurons	optimizer
0	Model1	0.730526	8	7,393,868	128	Adam
1	Model2	0.787368	10	3,306,188	128	Adam
2	Model3	0.772632	10	3,306,188	128	SGD
3	Model4	0.705263	17	834,252	128	Adam
4	Model5	0.755789	20	821,324	256	Adam
5	Model6	0.421053	17	26,200,716	256	Adam
6	Model7	0.795789	20	9,733,804	512	Adam
7	Model8	0.787368	20	894,028	256	Adam
8	Model9	0.722105	14	1,142,668	64	Adam

The model that performed best was Model4 with an overall accuracy of 0.83.

Model4 has 17 layers and trained 834,52 parameters. The fully connected layer had 128 neurons and the optimizer run was Adam.

The precision scores range from 0.48 to 0.95, indicating that the model's ability to correctly identify positive instances varies by class. For example, the model has high precision in identifying Charlock and Cleavers, while it has relatively lower precision in identifying Black-grass and Shepherds Purse.

The recall scores range from 0.50 to 0.97, indicating that the model's ability to correctly identify positive instances also varies by class. For example, the model has high recall in identifying Common Chickweed and Charlock, while it has relatively lower recall in identifying Black-grass and Shepherds Purse.

The F1-scores are a harmonic mean of precision and recall and range from 0.49 to 0.92. The F1-score provides a balance between precision and recall, indicating the overall performance of the model for each class.

Final Conclusions / Takeaways

The best performing model obtained has 4 Conv2D layers, 128 filters per layer, and 2 Dense layers with 256 neurons each. This model achieved an accuracy of 83% on the test set, which is a significant improvement from the baseline model.

We could further try to improve the model's performance by trying mostly:

• Increase the size of the training dataset by collecting more images or using data augmentation techniques to generate new images from the existing ones.

• Try using pre-trained models such as VGG, ResNet or Inception and fine-tune them on your dataset.

• Continue to experiment with different hyperparameters such as batch size, number of epochs, and regularization techniques to find the best combination for your model.

We should notice that building a good performing CNN model requires a lot of experimentation and fine-tuning.

Additionally keep in mind:

• Larger filter sizes may cause loss of spatial resolution, overfitting, and high computational cost.
• Having too many convolution layers can cause overfitting, vanishing gradients, cause longer training times and produce results that are difficult to interpret.
• Having too many neurons may cause overfitting and slow down the training process.
• Dropout can help prevent overfitting
• BatchNormalization can help improve performance by estabilizing the training process and allow for higher learning rates

Overall, our model performs well but it needs to be fine tuned for the classes that are most difficult to classify, in our case, Black grass.